Re: Async Image - ImageData conversion
On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Request for feedback: Streams API
Looks great! Seems very well thought through. The API seems large enough that it would be worth prototyping it and writing test applications to make sure it addresses key use cases before finalizing the spec. -Ken On Wed, Dec 4, 2013 at 8:27 AM, Feras Moussa feras.mou...@hotmail.com wrote: The editors of the Streams API have reached a milestone where we feel many of the major issues that have been identified thus far are now resolved and incorporated in the editors draft. The editors draft [1] has been heavily updated and reviewed the past few weeks to address all concerns raised, including: 1. Separation into two distinct types -ReadableByteStream and WritableByteStream 2. Explicit support for back pressure management 3. Improvements to help with pipe( ) and flow-control management 4. Updated spec text and diagrams for further clarifications There are still a set of bugs being tracked in bugzilla. We would like others to please review the updated proposal, and provide any feedback they may have (or file bugs). Thanks. -Feras [1] https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm
Re: File API - Progress - Question about Partial Blob data
On Fri, Aug 23, 2013 at 8:35 AM, Arun Ranganathan a...@mozilla.com wrote: On Aug 22, 2013, at 12:07 PM, Jonas Sicking wrote: I think you might have misunderstood my initial comment. I agree that the current partial data solution is not good. I think we should remove it. I'd really like other implementors to weigh in before we remove Partial Blob Data. Cc'ing folks who helped with it. Eric Urhane asked me to follow up on this thread on behalf of Gregg Tavares who unfortunately left Google. The current spec for partial blob data is too inefficient, because it accumulates all of the data since the beginning of the download. This is not what's desired for streaming downloads of large data sets. What's needed is a way to retrieve the data downloaded since the last query. Several web developers have asked about this recently as they're trying to stream ever larger 3D data sets into the browser. I think we should instead create a better solution in v2 of the API which is based on something other than FileReader and which has the ability to deliver data on the form of here's the data that was loaded since last notification. I agree that we should do a better way. Agreed. It would be really good to finally make progress in this area. It sounds like Microsoft's Streams API proposal at https://dvcs.w3.org/hg/streams-api/raw-file/tip/Overview.htm or tyoshino's Streams with Promises propsal at http://htmlpreview.github.io/?https://github.com/tyoshino/stream/blob/master/streams.html are two leading contenders. I personally don't care which flavor is chosen so long as things move forward. Microsoft's proposal does seem more fully fleshed out. (At least, it contains fewer instances of the word blah. :) ) -Ken
Re: exposing CANVAS or something like it to Web Workers
On Fri, Feb 8, 2013 at 9:14 AM, Travis Leithead travis.leith...@microsoft.com wrote: What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. Well, it's not a fleshed out proposal by any stretch, but you could imagine that an event could be used to signal that new frames were ready from the producer—then the main thread would know to pop. It sounds like this approach would require a bunch of new concepts -- a remote worker context, events signalled through it, etc. It would still be necessary to postMessage from the main thread back to the worker for flow control. Flow control is definitely necessary -- the producer can't just produce frames without any feedback about when they're actually consumed. We've had problems in Chrome's graphics pipeline in the past where lack of flow control led to slow and inconsistent frame rates. I'm excited about Gregg's proposal because it solves a lot of use cases that aren't currently addressed by CanvasProxy, using a couple of simple primitives that build upon others already in the platform (cross-document messaging and Transferables). How can we move forward so that user agents can experimentally implement these APIs? Ideally we'd prototype and experiment with them in some medium-sized applications before finalizing any specs in this area. Thanks, -Ken From: Gregg Tavares [mailto:g...@google.com] Sent: Friday, February 8, 2013 3:14 AM To: Travis Leithead Cc: Ian Hickson; Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers On Thu, Feb 7, 2013 at 10:46 PM, Travis Leithead travis.leith...@microsoft.com wrote: Having thought about this before, I wonder why we don’t use a producer/consumer model rather than a transfer of canvas ownership model? A completely orthogonal idea (just my rough 2c after reading Gregg’s proposal), is to have an internal frame buffer accessible via a WorkerCanvas API which supports some set of canvas 2d/3d APIs as appropriate, and can “push” a completed frame onto a stack in the internal frame buffer. Thus the worker can produce frames as fast as desired. On the document side, canvas gets a 3rd kind of context—a WorkerRemoteContext, which just offers the “pop” API to pop a frame from the internal frame buffer into the canvas. Then you just add some basic signaling events on both ends of the frame buffer and you’re good (as far as synchronizing the worker with the document). The producer (in the worker) is free to produce multiple frames in advance (if desired), while the consumer is able to pop frames when available. You could even have the framebuffer depth configurable. What would be the advantage? If you wanted to keep dom elements in sync with the canvas you'd still have to post something from the worker back to the main thread so the main thread would know to pop. From: Gregg Tavares [mailto:g...@google.com] Sent: Thursday, February 7, 2013 2:25 PM To: Ian Hickson Cc: Charles Pritchard; Web Applications Working Group WG Subject: Re: exposing CANVAS or something like it to Web Workers I put up a new proposal for canvas in workers http://wiki.whatwg.org/wiki/CanvasInWorkers Please take a look. This proposal comes from offline discussions with representatives from the various browsers as well as input from the Google Maps team. I can post a summary here if you'd like but it might be easier to read the wiki Looking forward to feedback. On Tue, Jan 8, 2013 at 10:50 AM, Ian Hickson i...@hixie.ch wrote: On Wed, 2 Jan 2013, Gregg Tavares (社ç~T¨) wrote: Another issue as come up and that is one of being able to synchronize updates of a canvas in worker with changes in the main page. For 2D, the intended solution is to just ship the ImageBitamp from the worker canvas to the main thread via a MessagePort and then render it on the canvas at the appropriate time. I don't know how you would do it for WebGL. Similarly, let's say you have 2 canvases and are rendering to both in a worker. Does context1.commit(); context2.commit(); guarantee you'll see both commits together? No, unfortunately not. There's no synchronisation between workers and the main thread (by design, to prevent any possibility of deadlocks), and there's not currently a batching API. However, if this becomes a common problem (which we can determine by seeing if we get bugs complaining about different parts of apps/games seeming to slide around or generally be slightly out of sync, or if we see a lot of authors shunting multiple ImageBitmap objects across MessagePort channels) we can always add an explicit batching API to make this kind of thing easy. Note that in theory, for 2D at least, shunting ImageBitmaps across threads can be as
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: exposing CANVAS or something like it to Web Workers
On Wed, May 16, 2012 at 1:11 PM, Gregg Tavares (勤) g...@google.com wrote: On Wed, May 16, 2012 at 12:42 PM, Kenneth Russell k...@google.com wrote: On Wed, May 16, 2012 at 12:30 PM, Gregg Tavares (勤) g...@google.com wrote: So how to take this forward? My #1 priority is to get WebGL in workers. Lots of developers have expressed a need for this from decoding compressed textures in a worker to offloading thousands of draw calls per frame to a worker. WebGL already defines sharing mechanisms so at least for the WebGL case I don't need to solve how to display anything back in the main worker. The worker can draw to a texture. The main page can use that texture. WebGL/OpenGL already define that relationship. But, to get WebGL in a worker I need a way to get a WebGLRenderingContext into a worker. Some Ideas *) Create context in main page, pass to worker Pros: . No new APIs or objects. Cons: . Transfer is messy. What happens to all the entry points and properties to the context object left in the main page? What happens to the cavnas parameter on the transferred context? How do you synchronize the main page setting canvas.width or canvas.height with the worker trying to render to it? *) Create a context directly in a Worker (or anywhere for that matter) As in var gl = new WebGLRenderingContext Pros: . ??? Cons: . requires defining how the backbuffer size is set. Maybe there is no backbuffer for a directly created WebGLRenderingContext? . if there is no backbuffer then using one of these contexts to draw into a texture or 2d context is problematic This alternative seems like the one that can be moved forward most easily, since essentially all of the changes would be within the WebGL spec. - Extend the spec to support context sharing. (Each WebGLRenderingContext points to an opaque WebGLContextGroup object.) - Define structured cloning semantics for WebGLContextGroup. - Add a constructor or factory method to WebGLRenderingContext allowing creation with a WebGLContextGroup. Contexts created in this manner would have no back buffer by default. (FBOs could still be used to do rendering with the context.) - Allow some or all of the WebGLObject types (textures, etc.) to be either copied during structured cloning or transferred. Then the worker can at least upload textures to the GPU completely asynchronously from the main thread, and inform the main thread via postMessage when they have finished uploading. That seems to be a tractable first step and one that would already have immediate benefits for developers. That problem I have with this path is the cons mentioned above. It's a dead end. With real contexts you can do this webglcontext1.texImage2D(..., webglcontext2.canvas); and this context2d.drawImage(webgl.canvas, ...) But with a context created with new WebGLRenderingContext you can't do anything like that because size of the backbuffer is not defined. Eventually we'll want to support operation like that and we'll likely end up with something like DrawingSurface or CanvasSurface or OffscreenCanvas. At that point this ability to go new WebGLRenderingContext will just be left over cruft. Also we'll have to define how to addEventListner for WebGLRenderContext for listening for lost context or for async context creation (yea, less important on workers). That stuff is currently not on WebGLRenderingContext. It seems like it should stay off if it. These are all good points, and I agree that the ideal fix would be to use the Canvas element (or DrawingSurface, etc.) from a worker, rather than doing a WebGL-specific hack. Given how long discussions have been ongoing on this topic, though, I think this may just be too large a single step to take. Rather than add a constructor to WebGLRenderingContext that would need to remain there permanently, perhaps a factory method with a vendor prefix could be added (e.g. WebGLRenderingContext.webkitCreateOffscreenContext, WebGLRenderingContext.mozCreateOffscreenContext). The long-term plan could be to remove the factory method once a better solution is reached, as opposed to removing the vendor prefixes. This would at least allow some applications to explore the use of workers with WebGL, see whether this addresses any existing issues, and expose new issues. -Ken -Ken *) Create an offscreen canvas like object minus HTMLElement parts Pros: . same or nearly the API as already exists for canvas . flexible. getContext can return different kinds of context. Maybe only webgl for now? Cons: . ???
Re: Should send() be able to take an ArrayBufferView?
On Wed, Apr 11, 2012 at 10:04 AM, Boris Zbarsky bzbar...@mit.edu wrote: Seems like right now passing a typed array to send() requires a bit of extra hoop-jumping to pass the .buffer instead, right? Is that desirable? It may be convenient to add an overload to send() (presumably on both XHR and WebSocket? Any others?) accepting ArrayBufferView. As pointed out elsewhere in this thread, ArrayBufferViews are a lightweight mechanism to refer to a sub-portion of an ArrayBuffer. All of the WebGL APIs which accept ArrayBuffer also accept ArrayBufferView, though the converse is not true. ArrayBufferView implies knowledge of the type of data being pointed to. In WebGL, some APIs accept only certain subclasses of ArrayBufferView, depending on other arguments to the function (readPixels, texImage2D, etc.). The reason XHR, WebSocket and other APIs refer only to ArrayBuffer is probably to insulate them from changes in the typed array specification. It seems that there is general agreement that the ArrayBuffer concept, of a completely opaque in-memory data container, is useful. It's clear that there is some disagreement with some of the other design decisions of the typed array specification, which might motivate leaving only the ArrayBuffer overload for the time being. Sending an ArrayBufferView would still have to use arraybuffer as the type of data. I don't think it would be a good idea to try to instantiate the same subclass of ArrayBufferView on the receiving side. -Ken
Re: Should send() be able to take an ArrayBufferView?
On Wed, Apr 11, 2012 at 2:47 PM, Charles Pritchard ch...@jumis.com wrote: On 4/11/2012 2:41 PM, Kenneth Russell wrote: On Wed, Apr 11, 2012 at 10:04 AM, Boris Zbarskybzbar...@mit.edu wrote: Seems like right now passing a typed array to send() requires a bit of extra hoop-jumping to pass the .buffer instead, right? Is that desirable? It may be convenient to add an overload to send() (presumably on both XHR and WebSocket? Any others?) accepting ArrayBufferView. As pointed It's convenient. xhr.send(view); // shorthand xhr.send(view.buffer.slice(view.byteOffset, view.byteOffset+view.byteLength)); // longhand. Kenneth, Can we get a voice from MS? They've been supporting typed arrays in IE10 for awhile. If we're going to have this method, I'd really like to see it in IE10. I have no control over that -- I believe there are representatives from Microsoft that monitor this list and hope they will voice an opinion. -Ken Boris, How do we feature test for support of the shorthand method? -Charles
Re: Should send() be able to take an ArrayBufferView?
On Wed, Apr 11, 2012 at 2:48 PM, Boris Zbarsky bzbar...@mit.edu wrote: On 4/11/12 5:41 PM, Kenneth Russell wrote: Sending an ArrayBufferView would still have to use arraybuffer as the type of data. I don't think it would be a good idea to try to instantiate the same subclass of ArrayBufferView on the receiving side. I'm not sure what this means... What I mean is that if somehow a browser were on the receiving end of one of these messages, the type of the incoming message should still be arraybuffer. For XHR.send(), sending an ArrayBufferView should take the byte array that the ArrayBufferView is mapping, and send that. It's possible to achieve the same thing now with some hoop jumping involving a possible buffer copy; I'm just saying we should remove the need for that hoop jumping. Agree that these should be the semantics. I haven't looked at WebSocket in enough detail to comment intelligently on it. I haven't really either, but if there were some peer-to-peer support, then the receiving peer should still get an ArrayBuffer even if the sender sent an ArrayBufferView. -Ken
Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal
On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote: You can always call close() yourself, but Blob.close() should use the neuter mechanism already there, not make up a new one. Blobs aren't transferable, there is no existing mechanism that applies to them. Adding a blob.close() method is independent of making blob's transferable, the former is not prerequisite on the latter. There is an existing mechanism for closing objects. It's called neutering. Blob.close should use the same terminology, whether or not the object is a Transferable. On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote: I would be hesitant to impose a close() method on all future Transferable types. Why? All Transferable types must define how to neuter objects; all close() does is trigger it. I don't think adding one to ArrayBuffer would be a bad idea but I think that ideally it wouldn't be necessary. On memory constrained devices, it would still be more efficient to re-use large ArrayBuffers rather than close them and allocate new ones. That's often not possible, when the ArrayBuffer is returned to you from an API (eg. XHR2). This sounds like a good idea. As you pointed out offline, a key difference between Blobs and ArrayBuffers is that Blobs are always immutable. It isn't necessary to define Transferable semantics for Blobs in order to post them efficiently, but it was essential for ArrayBuffers. No new semantics need to be defined; the semantics of Transferable are defined by postMessage and are the same for all transferable objects. That's already done. The only thing that needs to be defined is how to neuter an object, which is what Blob.close() has to define anyway. Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future large, structured clonable objects to all be released with the same mechanisms: either pass them in the transfer argument to a postMessage call, or use the consistent, identical close() method inherited from Transferable. This allows developers to think of the transfer list as a list of objects which won't be needed after the postMessage call. It doesn't matter that the underlying optimizations are different; the visible side-effects are identical (the object can no longer be accessed). Closing an object, and neutering it because it was transferred to a different owner, are different concepts. It's already been demonstrated that Blobs, being read-only, do not need to be transferred in order to send them efficiently from one owner to another. It's also been demonstrated that Blobs can be resource intensive and that an explicit closing mechanism is needed. I believe that we should fix the immediate problem and add a close() method to Blob. I'm not in favor of adding a similar method to ArrayBuffer at this time and therefore not to Transferable. There is a high-level goal to keep the typed array specification as minimal as possible, and having Transferable support leak in to the public methods of the interfaces contradicts that goal. -Ken
Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal
On Wed, Mar 7, 2012 at 12:02 PM, Charles Pritchard ch...@jumis.com wrote: On Mar 7, 2012, at 11:38 AM, Kenneth Russell k...@google.com wrote: On Tue, Mar 6, 2012 at 6:29 PM, Glenn Maynard gl...@zewt.org wrote: On Tue, Mar 6, 2012 at 4:24 PM, Michael Nordman micha...@google.com wrote: You can always call close() yourself, but Blob.close() should use the neuter mechanism already there, not make up a new one. Blobs aren't transferable, there is no existing mechanism that applies to them. Adding a blob.close() method is independent of making blob's transferable, the former is not prerequisite on the latter. There is an existing mechanism for closing objects. It's called neutering. Blob.close should use the same terminology, whether or not the object is a Transferable. On Tue, Mar 6, 2012 at 4:25 PM, Kenneth Russell k...@google.com wrote: I would be hesitant to impose a close() method on all future Transferable types. Why? All Transferable types must define how to neuter objects; all close() does is trigger it. I don't think adding one to ArrayBuffer would be a bad idea but I think that ideally it wouldn't be necessary. On memory constrained devices, it would still be more efficient to re-use large ArrayBuffers rather than close them and allocate new ones. That's often not possible, when the ArrayBuffer is returned to you from an API (eg. XHR2). This sounds like a good idea. As you pointed out offline, a key difference between Blobs and ArrayBuffers is that Blobs are always immutable. It isn't necessary to define Transferable semantics for Blobs in order to post them efficiently, but it was essential for ArrayBuffers. No new semantics need to be defined; the semantics of Transferable are defined by postMessage and are the same for all transferable objects. That's already done. The only thing that needs to be defined is how to neuter an object, which is what Blob.close() has to define anyway. Using Transferable for Blob will allow Blobs, ArrayBuffers, and any future large, structured clonable objects to all be released with the same mechanisms: either pass them in the transfer argument to a postMessage call, or use the consistent, identical close() method inherited from Transferable. This allows developers to think of the transfer list as a list of objects which won't be needed after the postMessage call. It doesn't matter that the underlying optimizations are different; the visible side-effects are identical (the object can no longer be accessed). Closing an object, and neutering it because it was transferred to a different owner, are different concepts. It's already been demonstrated that Blobs, being read-only, do not need to be transferred in order to send them efficiently from one owner to another. It's also been demonstrated that Blobs can be resource intensive and that an explicit closing mechanism is needed. I believe that we should fix the immediate problem and add a close() method to Blob. I'm not in favor of adding a similar method to ArrayBuffer at this time and therefore not to Transferable. There is a high-level goal to keep the typed array specification as minimal as possible, and having Transferable support leak in to the public methods of the interfaces contradicts that goal. I think there's broad enough consensus amongst vendors to table the discussion about adding close to Transferable. Would you please let me know why ypu believe ArrayBuffer should not have a close method? I would like some clarity here. The Typed Array spec would not be cluttered by the addition of a simple close method. It's certainly a matter of opinion -- but while it's only the addition of one method, it changes typed arrays' semantics to be much closer to manual memory allocation than they currently are. It would be a further divergence in behavior from ordinary ECMAScript arrays. The TC39 working group, I have heard, is incorporating typed arrays into the language specification, and for this reason I believe extreme care is warranted when adding more functionality to the typed array spec. The spec can certainly move forward, but personally I'd like to check with TC39 on semantic changes like this one. That's the rationale behind my statement above about preferring not to add this method at this time. -Ken I work much more with ArrayBuffer than Blob. I suspect others will too as they progress with more advanced and resource intensive applications. What is the use-case distinction between close of immutable blob and close of a mutable buffer? -Charles
Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal
On Wed, Mar 7, 2012 at 1:00 PM, Charles Pritchard ch...@jumis.com wrote:
On 3/7/12 12:34 PM, Kenneth Russell wrote:
On Wed, Mar 7, 2012 at 12:02 PM, Charles Pritchardch...@jumis.com
wrote:
On Mar 7, 2012, at 11:38 AM, Kenneth Russellk...@google.com wrote:
I believe that we should fix the immediate problem and add a close()
method to Blob. I'm not in favor of adding a similar method to
ArrayBuffer at this time and therefore not to Transferable. There is a
high-level goal to keep the typed array specification as minimal as
possible, and having Transferable support leak in to the public
methods of the interfaces contradicts that goal.
I think there's broad enough consensus amongst vendors to table the
discussion about adding close to Transferable.
Would you please let me know why ypu believe ArrayBuffer should not have
a close method?
I would like some clarity here. The Typed Array spec would not be
cluttered by the addition of a simple close method.
It's certainly a matter of opinion -- but while it's only the addition
of one method, it changes typed arrays' semantics to be much closer to
manual memory allocation than they currently are. It would be a
further divergence in behavior from ordinary ECMAScript arrays.
The TC39 working group, I have heard, is incorporating typed arrays
into the language specification, and for this reason I believe extreme
care is warranted when adding more functionality to the typed array
spec. The spec can certainly move forward, but personally I'd like to
check with TC39 on semantic changes like this one. That's the
rationale behind my statement above about preferring not to add this
method at this time.
Searching through the net tells me that this has been a rumor for years.
Regardless of rumors I have talked to multiple members of TC39 who
have clearly stated it is being incorporated into ES6 Harmony.
I agree with taking extreme care -- so let's isolate one more bit of
information:
Is ArrayBuffer being proposed for TC39 incorporation, or is it only the
Typed Arrays? The idea here is to alter ArrayBuffer, an object which can be
neutered via transfer map. It seems a waste to have to create a Worker to
close down buffer views.
Both ArrayBuffer and the typed array views will be incorporated.
Will TC39 have anything to say about the neuter concept and/or Web
Messaging?
This is an excellent question and one which I've also posed to TC39. I
don't see how the language spec could reference these concepts. I'm
guessing that this is an area that TC39 hasn't yet figured out,
either.
Again, I'm bringing this up for the same practical experience that
Blob.close() was brought up. I do appreciate that read/write allocation is a
separate semantic from write-once/read-many allocation.
I certainly don't want to derail the introduction of Typed Array into TC39.
I don't want to sit back for two years either, while the ArrayBuffer object
is in limbo.
Understood and appreciated.
If necessary, I'll do some of the nasty test work of creating a worker
simply to destroy buffers, and report back on it.
var worker = new Worker('trash.js');
worker.postMessage(null,[bufferToClose]);
worker.close();
vs.
bufferToClose.close();
I doubt that that will work. Garbage collection will still need to run
in the worker's JavaScript context in order for the transferred
ArrayBuffer to be cleaned up, and I doubt that happens eagerly upon
shutdown of the worker. Would be happy to be proven wrong.
If you prototype adding ArrayBuffer.close() in your open source
browser of choice and report back on significant efficiency
improvements in a real-world use case, that would be valuable
feedback.
-Ken
Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal
On Tue, Mar 6, 2012 at 12:04 PM, Greg Billock gbill...@google.com wrote: On Mon, Mar 5, 2012 at 6:46 PM, Charles Pritchard ch...@jumis.com wrote: On 3/5/2012 5:56 PM, Glenn Maynard wrote: On Mon, Mar 5, 2012 at 7:04 PM, Charles Pritchard ch...@jumis.com wrote: Do you see old behavior working something like the following? var blob = new Blob(my new big blob); var keepBlob = blob.slice(); destination.postMessage(blob, '*', [blob]); // is try/catch needed here? You don't need to do that. If you don't want postMessage to transfer the blob, then simply don't include it in the transfer parameter, and it'll perform a normal structured clone. postMessage behaves this way in part for backwards-compatibility: so exactly in cases like this, we can make Blob implement Transferable without breaking existing code. See http://dev.w3.org/html5/postmsg/#posting-messages and similar postMessage APIs. Web Intents won't have a transfer map argument. http://dvcs.w3.org/hg/web-intents/raw-file/tip/spec/Overview.html#widl-Intent-data For the Web Intents structured cloning algorithm, Web Intents would be inserting into step 3: If input is a Transferable object, add it to the transfer map. http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#internal-structured-cloning-algorithm Then Web Intents would move the first section of the structured cloning algorithm to follow the internal cloning algorithm section, swapping their order. http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#safe-passing-of-structured-data That's my understanding. We've been discussing the merits of this approach vs using a transfer array argument. There's a lot to like about this alternative -- it conserves arguments and looks simpler than the transfer map, as well as not having the headaches of whether you can do (null, [port]) or (port, [port]) and concerns like that. The advantage of using the transfer map param is that it is more contiguous with existing practice. We'd kind of hoped that this particular debate was finalized before we got to the point of needing to make a decision, so we bluffed and left it out of the web intents spec draft. :-) At this point, I'm leaning toward needing to add a transfer map parameter, and then dealing with that alongside other uses, given the state of thinking on Transferables support and the need to make this pretty consistent across structure clone invocations. I do think that complexity might be better solved by the type system (i.e. a new Transferable(ArrayBuffer)), which would require a different developer mechanic to set up clone vs transfer, but would relieve complexity in the invocation of structured clone itself: transferables could just always transfer transparently. I don't know if, given current practice with MessagePort, that kind of solution is available. A change like this would be feasible as long as it doesn't break compatibility. In other words, the current Transferable array would still need to be supported, but Transferable instances (or perhaps instances of some other type) wrapping another Transferable object would also express the intent. The current API for Transferable and postMessage was informed by the realization that the previous sequenceMessagePort argument to postMessage was essentially already expressing the Transferable concept. I'm not familiar with the Web Intents API, but at first glance it seems feasible to overload the constructor, postResult and postFailure methods to support passing a sequenceTransferable as the last argument. This would make the API look more like postMessage and avoid adding more transfer semantics. Is that possible? Something like this may be necessary if Blob were a Transferable: var keepBlob = blob.slice(); var intent = new Intent(-x-my-intent, blob); navigator.startActivity(intent, callback); And we might have an error on postMessage stashing it in the transfer array if it's not a Transferable on an older browser. Example of how easy the neutered concept applies to Transferrable: var blob = new Blob(my big blob); blob.close(); I like the idea of having Blob implement Transferrable and adding close to the Transferrable interface. File.close could have a better relationship with the cache and/or locks on data. I'm not sure that adding close() to Transferable is a good idea. Not all Transferable types may want to support that explicit operation. What about adding close() to Blob, and having the neutering operation on Blob be defined to call close() on it? -Ken Some history on Transferrable and structured clones: Note: MessagePort does have a close method and is currently the only Transferrable mentioned in WHATWG: http://www.whatwg.org/specs/web-apps/current-work/multipage/common-dom-interfaces.html#transferable-objects ArrayBuffer is widely implemented. It was the second item to implement
Re: Transferable and structured clones, was: Re: [FileAPI] Deterministic release of Blob proposal
On Tue, Mar 6, 2012 at 1:31 PM, Arun Ranganathan aranganat...@mozilla.com wrote: Ken, I'm not sure that adding close() to Transferable is a good idea. Not all Transferable types may want to support that explicit operation. What about adding close() to Blob, and having the neutering operation on Blob be defined to call close() on it? Specifically, you think this is not something ArrayBuffer should inherit? If it's also a bad idea for MessagePort, then those are really our only two use cases of Transferable right now. I'm happy to create something like a close() on Blob. MessagePort already defines a close() operation, so there's really no question of whether its presence is a good or bad idea there. A close() operation seems necessary in networking style APIs. I would be hesitant to impose a close() method on all future Transferable types. I don't think adding one to ArrayBuffer would be a bad idea but I think that ideally it wouldn't be necessary. On memory constrained devices, it would still be more efficient to re-use large ArrayBuffers rather than close them and allocate new ones. On Tue, Mar 6, 2012 at 1:34 PM, Michael Nordman micha...@google.com wrote: Sounds like there's a good case for an explicit blob.close() method independent of 'transferable'. Separately defining blobs to be transferrable feels like an unneeded complexity. A caller wishing to neuter after sending can explicit call .close() rather than relying on more obscure artifacts of having also put the 'blob' in a 'transferrable' array. This sounds like a good idea. As you pointed out offline, a key difference between Blobs and ArrayBuffers is that Blobs are always immutable. It isn't necessary to define Transferable semantics for Blobs in order to post them efficiently, but it was essential for ArrayBuffers. -Ken
Re: String to ArrayBuffer
The StringEncoding proposal is the best path forward because it provides correct behavior in all cases. Adding String conversions directly to the typed array spec will introduce dependencies that are strongly undesirable, and make it much harder to implement the core spec. Hopefully Josh can provide an update on how the StringEncoding proposal is going. -Ken On Wed, Jan 11, 2012 at 3:05 PM, Charles Pritchard ch...@jumis.com wrote: On 1/11/2012 2:49 PM, James Robinson wrote: On Wed, Jan 11, 2012 at 2:45 PM, Charles Pritchard ch...@jumis.com wrote: Currently, we can asynchronously use BlobBuilder with FileReader to get an array buffer from a string. We can of course, use code to convert String.fromCharCode into a Uint8Array, but it's ugly. The StringEncoding proposal seems a bit much for most web use: http://wiki.whatwg.org/wiki/StringEncoding All we really ever do is work on DOMString, and that's covered by UTF8. DOMString is not UTF8 or necessarily unicode. It's a sequence of 16 bit integers and a length. To clarify, I'd want ArrayBuffer(DOMString) to work with unicode and throw an error if the DOMString is not valid unicode. This is consistent with other Web Apps APIs. For feature detection, the method should be wrapped in a try-catch block anyway. -Charles
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Fri, Jun 24, 2011 at 4:27 PM, Kenneth Russell k...@google.com wrote: On Fri, Jun 24, 2011 at 3:43 PM, Ian Hickson i...@hixie.ch wrote: On Fri, 24 Jun 2011, Kenneth Russell wrote: Slightly larger issue. In the typed array spec, views like Float32Array refer to an ArrayBuffer instance. It's desired to be able to transfer multiple views of the same ArrayBuffer in the same postMessage call. Currently, because each Transferable is transferred independently, transferring the first view will cause the view and underlying ArrayBuffer to be neutered, so upon encountering the second view, an exception will be thrown since its ArrayBuffer was already transferred. The views shouldn't be Transferable. Only the buffer should be. The views should continue to have the behaviour you had described before, where they recurse to clone their buffer then just clone the view. Since the buffers would already be transferred (the transfering happens before the cloning), it'll all just work. Thanks, I think I see how this is supposed to work now. I'll rewrite the relevant sections of the typed array spec soon and ping you or post again if it looks like there are any other issues. I apologize for how long this took, but the editor's draft of the typed array spec has finally been updated with these changes. I found it a little difficult to specify the desired behavior; please let me know if you see a better or simpler way. https://www.khronos.org/registry/typedarray/specs/latest/ -Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Thu, Jun 23, 2011 at 4:52 PM, Ian Hickson i...@hixie.ch wrote: On Tue, 21 Jun 2011, Ian Hickson wrote: How about we just make postMessage() take the object to clone in the first argument, an array of objects to transfer in the second; on the other side, the author receives the object cloned, with anything listed in the array and in the structured data being transferred instead of cloned, and, in addition, in the event.ports array, a list of the ports that were given in the transfer array. This has the advantage of being completely backwards-compatible. So for example, on the sending side: postMessage(['copied', copiedArrayBuffer, transferredArrayBuffer, transferredPort1], // could be an object too [transferredArrayBuffer, transferredPort1, transferredPort2]); ...on the receiving side, the event has data == ['copied', newCopiedArrayBuffer, newTransferredArrayBuffer, newTransferredPort1]; ports == [newTransferredPort1, newTransferredPort2]; It's not going to win any design awards, but I think that boat has sailed for this particular API anyway, given the contraints we're operating under. Since no serious problems were raised with this and it seems to address all the constraints, I've now specced this. One edge case that wasn't mentioned above is what happens when a non-port Transferable object is in the second list but not the first. I defined it such that it still gets cloned (and thus the original is neutered), but it is then discarded. (That definition more or less fell out of the way one'd have to implement this to make it simple to understand, but I'm happy to do it a different way if there's a good reason to.) http://html5.org/tools/web-apps-tracker?from=6272to=6273 Thanks, this looks great! Minor issue: there are a few places where transfered should be transferred. Slightly larger issue. In the typed array spec, views like Float32Array refer to an ArrayBuffer instance. It's desired to be able to transfer multiple views of the same ArrayBuffer in the same postMessage call. Currently, because each Transferable is transferred independently, transferring the first view will cause the view and underlying ArrayBuffer to be neutered, so upon encountering the second view, an exception will be thrown since its ArrayBuffer was already transferred. This could happen with any Transferable object that has a reference to another Transferable, so I don't think it's a problem overly specific to typed arrays. The way this was addressed in the current typed array strawman proposals was to split the transfer operation into two stages: cloning of, and closing of, the original object. First, all transferable objects were cloned, and then they all were closed. See http://www.khronos.org/registry/typedarray/specs/latest/#9.2 . This can be addressed in the current spec by stating in http://www.whatwg.org/specs/web-apps/current-work/multipage/urls.html#transferable-objects that the transfer map is provided to the steps defined by the type of the object in question. That way, transferring of a particular object can update the map during the transfer operation, possibly adding more associations to it. The map will provide enough state to allow transfer of dependent transferable objects. Thoughts? Should I file a bug about this? kbr: Feel free to ping me if you need advice on how to use this with ArrayBuffer in the short term. On the long term I'm happy to just spec this in the same spec as the rest of this stuff. Thanks. In the long run I'd be very happy to have the semantics for ArrayBuffer and views defined in this spec. In the short term, I'd like to prototype this first and get some experience with it. -Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Tue, Jun 21, 2011 at 11:43 PM, Glenn Maynard gl...@zewt.org wrote:
On Wed, Jun 22, 2011 at 1:57 AM, David Levin le...@chromium.org wrote:
Let's say the call doesn't throw when given a type B that isn't
transferrable.
Let's also say some later changes the javascript code and uses B after the
postMessage call.
Everything work. No throw is done and B isn't gutted because it isn't
transferrable.
Throwing for unsupported objects will break common, reasonable uses, making
everyone jump hoops to use transfer. Not throwing will only break code that
seriously misuses the API, by listing objects for transfer and then
continuing to use the object.
Anyway, if this exception is thrown, everyone's going to use a helper like
this:
function filterTransferrable(list) {
var ret = [];
for(var i = 0; i list.length; ++i) {
if(list[i] instanceof Transferrable)
ret.push(list[i]);
}
return ret;
}
postMessage([A, B], filterTransferrable([A, B]));
... which will trigger the case you describe anyway.
The structured cloning algorithm itself will throw an exception if it
encounters an object type it doesn't know how to handle, in particular
a host object. See
http://www.whatwg.org/specs/web-apps/current-work/multipage/urls.html#safe-passing-of-structured-data
. As browsers evolve, it will be host object types that are promoted
to transferable status. Therefore, applications that are written for
browser version N+1 which supports transferring host object type Foo
will throw an exception on browser version N when sending a Foo via
postMessage simply because of its presence in the object graph,
regardless of its presence in the transfer array.
For this reason, and because I also prefer fail-fast APIs, I agree
that unsupported objects in the transfer array should raise an
exception.
I doubt that authors will write a filter for the transferable array as
you suggest, since filtering of the object graph would be necessary
too. Rather, I think that applications will be written for browsers
that support transferring host objects of a particular type.
The current proposal sounds good to me.
-Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Thu, Jun 9, 2011 at 10:54 PM, Travis Leithead travis.leith...@microsoft.com wrote: Honestly, there’s something about this whole discussion that just doesn’t feel right. I looks like we’re trying to graft-in this new concept of transfer of ownership into the existing postMessage semantics (i.e., object cloning). Any way I try to make it work, it just looks like peaches grafted into an apple tree. What happened to Jonas’ other proposal about a new API? I’d like to direct some mental energy into that proposal. Complexity comes in many forms and shapes. I much more like the idea of explicit APIs that make it clear what happens and make it hard to shoot yourself in the foot. Yes, it can involve more typing, but if it results in more resilient code which contains fewer subtle bugs, then I think we have designed the API well. / Jonas Ex: void postMessageAndTransfer([in] any transferOwnershipToDestination…); We’re only talking about a scenario that makes sense primarily for Web Workers and applies only to certain types like ArrayBuffer, CanvasPixelArray+ImageData, Blob, File, etc., that have large underlying memory buffers. We don’t really need to support JavaScript objects, arrays, complex graphs, etc. at all with the new API (and since the current proposal requires the web developer to make an explicit list anyway for the 2nd param to post message, it’s no _more_ work to do the same for a new API). We could even try to graft MessagePorts into this API, but why? MessagePorts are unique in function compared to the other objects we are discussing for transfer of ownership (e.g., they facilitate further messaging and can’t be re-posted once they are cloned once), and they already have well-defined behavior in MessageEvents and SharedWorkers. I propose keeping postMessage exactly as it is. Let’s eliminate the potential compatibility issues. Let’s not re-write the existing specs (that feels like going backwards, not forwards). For transfer of ownership, let’s bring this capability on-line through a new API, for the specific scenario where it makes sense (Web Workers) and not pollute the current postMessage concepts (object graph cloning and port-passing). Travis, I disagree with your statement that MessagePorts are unique in function compared to the other objects we are discussing for transfer of ownership. Cloning a MessagePort per http://dev.w3.org/html5/postmsg/#clone-a-port is *exactly* transferring its ownership to the other side. The reason that a MessagePort object can only be cloned once is that its ownership has been transferred. There is no restriction in the current specification preventing the cloned port from being transferred to a new owner via postMessage. The current proposal on the table is 100% backward compatible in signature and semantics, and is an elegant generalization of the slightly over-specialized MessagePort mechanism into the desired transfer of ownership mechanism. In any other API I would personally want exactly postMessage's capability of sending full JavaScript object graphs over the wire, while still being able to transfer ownership of some of the objects contained within, to be able to add some structure to the messages being sent. I would not want to artificially restrict the API to only be able to send certain types of objects. -Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
My understanding is that we have reached a proposal which respecifies
the ports argument to postMessage as an array of objects to
transfer, in such a way that we:
- Maintain 100% backward compatibility
- Enhance the ability to pass MessagePorts, so that the object graph
can refer to them as well
- Allow more object types to participate in transfer of ownership in the future
To the best of my knowledge there are no active points of
disagreement. I think we are only waiting for general consensus from
all interested parties that this is the desired step to take.
If it is, I would be happy to draft proposed edits to the associated
specs; there are several, and the edits may be somewhat involved. I'd
also be happy to share the work with Ian or anyone else.
I don't know the various processes for web specs, but the Web
Messaging spec will definitely need to be updated if we decide to move
in this direction.
-Ken
On Wed, Jun 8, 2011 at 4:30 AM, Arthur Barstow art.bars...@nokia.com wrote:
Now that the responses on this thread have slowed, I would appreciate if the
participants would please summarize where they think we are on this issue,
e.g. the points of agreement and disagreement, how to move forward, etc.
Also, coming back to the question in the subject (and I apologize if my
premature subject change caused any confusion or problems), since we have an
open CfC (ends June 9 [1]) to publish a Candidate Recommendation of Web
Messaging, is the Messaging spec going to need to change to address the
issues raised in this thread?
-Art Barstow
[1] http://lists.w3.org/Archives/Public/public-webapps/2011AprJun/0797.html
On Jun/3/2011 8:47 PM, ext Kenneth Russell wrote:
On Fri, Jun 3, 2011 at 4:15 PM, Andrew Wilsonatwil...@google.com wrote:
On Fri, Jun 3, 2011 at 3:23 PM, Glenn Maynardgl...@zewt.org wrote:
On Fri, Jun 3, 2011 at 5:15 PM, Andrew Wilsonatwil...@google.com
wrote:
significant motivation. The stated motivations for breaking this API
don't
seem compelling to me given the existence of backwards-compatible
alternatives.
This proposal is backwards-compatible. If the argument is an array,
nothing changes, so postMessage(..., [ports]) is equivalent to
postMessage(..., {ports: [ports]}). (The array-only approach can be
done compatibly, too; the object version is just an alternative to
that.) What's backwards-incompatible?
Ah, I missed that piece (to be honest, I haven't been following this
discussion in every detail - I only chimed in because of Jonas' request
for
implementation feedback).
For anyone not looking closely at the IDL while reading this, this
means deprecating (for whatever value deprecate has on the web) the
ports array in MessageEvent--not the ports parameter to postMessage
(that's a sequence).
Does this affect the API for the SharedWorker onconnect message as well?
Good point; that might inform not deprecating the ports array in
MessageEvent, but leaving it as is.
-Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
I prefer continuing to use an array for several reasons: simpler
syntax, better type checking at the Web IDL level, and fewer
ECMAScript-specific semantics.
-Ken
On Wed, Jun 8, 2011 at 2:29 PM, David Levin le...@chromium.org wrote:
On Wed, Jun 8, 2011 at 2:24 PM, Kenneth Russell k...@google.com wrote:
My understanding is that we have reached a proposal which respecifies
the ports argument to postMessage as an array of objects to
transfer, in such a way that we:
Array or object? (by object I mean: {transfer: [arrayBuffer1], ports:
[port]})
- Maintain 100% backward compatibility
- Enhance the ability to pass MessagePorts, so that the object graph
can refer to them as well
- Allow more object types to participate in transfer of ownership in the
future
To the best of my knowledge there are no active points of
disagreement. I think we are only waiting for general consensus from
all interested parties that this is the desired step to take.
If it is, I would be happy to draft proposed edits to the associated
specs; there are several, and the edits may be somewhat involved. I'd
also be happy to share the work with Ian or anyone else.
I don't know the various processes for web specs, but the Web
Messaging spec will definitely need to be updated if we decide to move
in this direction.
-Ken
On Wed, Jun 8, 2011 at 4:30 AM, Arthur Barstow art.bars...@nokia.com
wrote:
Now that the responses on this thread have slowed, I would appreciate if
the
participants would please summarize where they think we are on this
issue,
e.g. the points of agreement and disagreement, how to move forward, etc.
Also, coming back to the question in the subject (and I apologize if my
premature subject change caused any confusion or problems), since we
have an
open CfC (ends June 9 [1]) to publish a Candidate Recommendation of Web
Messaging, is the Messaging spec going to need to change to address the
issues raised in this thread?
-Art Barstow
[1]
http://lists.w3.org/Archives/Public/public-webapps/2011AprJun/0797.html
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Wed, Jun 8, 2011 at 2:39 PM, David Levin le...@chromium.org wrote:
On Wed, Jun 8, 2011 at 2:33 PM, Kenneth Russell k...@google.com wrote:
I prefer continuing to use an array for several reasons: simpler
syntax, better type checking at the Web IDL level, and fewer
ECMAScript-specific semantics.
An array makes it harder to do future modifications.
Possibly, but it makes the design of this modification cleaner.
Also with the array, how does Enhance the ability to pass MessagePorts, so
that the object graph can refer to them as well work? Specifically,
consider an array that contains [arrayBuffer1, port1]. Is port1 something in
the object graph or a port to be transfer as before?
In order to maintain backward compatibility, the clone of port1 would
show up in the ports attribute of the MessageEvent on the other
side. Additionally, during the structured clone of the object graph,
any references to port1 would be updated to point to the clone of
port1. (The latter is new behavior, and brings MessagePorts in line
with the desired transfer-of-ownership semantics.)
All other objects in the array (which, as Ian originally proposed,
would implement some interface like Transferable for better Web IDL
type checking) would simply indicate objects in the graph to be
transferred rather than copied.
Note: it would still be possible to evolve the API to transfer all
objects of a certain type. We would just need to change the type of
the ports or transfer array from Transferable[] to any[] and spec
what happens when a constructor function is placed in the array.
-Ken
dave
-Ken
On Wed, Jun 8, 2011 at 2:29 PM, David Levin le...@chromium.org wrote:
On Wed, Jun 8, 2011 at 2:24 PM, Kenneth Russell k...@google.com wrote:
My understanding is that we have reached a proposal which respecifies
the ports argument to postMessage as an array of objects to
transfer, in such a way that we:
Array or object? (by object I mean: {transfer: [arrayBuffer1], ports:
[port]})
- Maintain 100% backward compatibility
- Enhance the ability to pass MessagePorts, so that the object graph
can refer to them as well
- Allow more object types to participate in transfer of ownership in
the
future
To the best of my knowledge there are no active points of
disagreement. I think we are only waiting for general consensus from
all interested parties that this is the desired step to take.
If it is, I would be happy to draft proposed edits to the associated
specs; there are several, and the edits may be somewhat involved. I'd
also be happy to share the work with Ian or anyone else.
I don't know the various processes for web specs, but the Web
Messaging spec will definitely need to be updated if we decide to move
in this direction.
-Ken
On Wed, Jun 8, 2011 at 4:30 AM, Arthur Barstow art.bars...@nokia.com
wrote:
Now that the responses on this thread have slowed, I would appreciate
if
the
participants would please summarize where they think we are on this
issue,
e.g. the points of agreement and disagreement, how to move forward,
etc.
Also, coming back to the question in the subject (and I apologize if
my
premature subject change caused any confusion or problems), since we
have an
open CfC (ends June 9 [1]) to publish a Candidate Recommendation of
Web
Messaging, is the Messaging spec going to need to change to address
the
issues raised in this thread?
-Art Barstow
[1]
http://lists.w3.org/Archives/Public/public-webapps/2011AprJun/0797.html
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Wed, Jun 8, 2011 at 6:14 PM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jun 8, 2011 at 4:27 PM, Kenneth Russell k...@google.com wrote: On Wed, Jun 8, 2011 at 2:39 PM, David Levin le...@chromium.org wrote: On Wed, Jun 8, 2011 at 2:33 PM, Kenneth Russell k...@google.com wrote: I prefer continuing to use an array for several reasons: simpler syntax, better type checking at the Web IDL level, and fewer ECMAScript-specific semantics. An array makes it harder to do future modifications. Possibly, but it makes the design of this modification cleaner. Also with the array, how does Enhance the ability to pass MessagePorts, so that the object graph can refer to them as well work? Specifically, consider an array that contains [arrayBuffer1, port1]. Is port1 something in the object graph or a port to be transfer as before? In order to maintain backward compatibility, the clone of port1 would show up in the ports attribute of the MessageEvent on the other side. Additionally, during the structured clone of the object graph, any references to port1 would be updated to point to the clone of port1. (The latter is new behavior, and brings MessagePorts in line with the desired transfer-of-ownership semantics.) All other objects in the array (which, as Ian originally proposed, would implement some interface like Transferable for better Web IDL type checking) would simply indicate objects in the graph to be transferred rather than copied. This all sounds great to me, but I think we should additionally make the 'ports' attribute on the MessageEvent interface deprecated. The only use case for it is to support existing code which doesn't pass ports in the object graph but rather only in the array in the second argument (i.e. the formerly ports argument). That's not quite correct; as was pointed out earlier in the thread, SharedWorkerGlobalScope's onconnect message relies on receiving the message port with which to communicate to the outside world in the zeroth element of the ports array. See step 7.7.5 in http://www.whatwg.org/specs/web-workers/current-work/#shared-workers-and-the-sharedworker-interface . Thinking about this more, that port could be sent as the data attribute of the event instead of the empty string. Then the ports attribute on MessageEvent could be safely deprecated. -Ken By deprecating it, I mean either: 1. Mark it, using prose, as deprecated in the specification. 2. Remove it from the specification but allow existing implementations of it to keep it as long as they feel needed to retain compatibility with existing code. / Jonas
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Fri, Jun 3, 2011 at 9:46 AM, Glenn Maynard gl...@zewt.org wrote:
On Fri, Jun 3, 2011 at 11:12 AM, Dmitry Lomov dslo...@google.com wrote:
a) Recursive transfer lists. Allow arbitrary objects, not only ArrayBuffers,
to appear in transfer lists. ArrayBuffers that are under objects in
transfer lists are transferred, others are cloned.
This again causes the same forwards-compatibility problem. If you do this:
frame = { video: canvasPixelArrayInstance, histogram: arrayBuffer } }
postMessage({ frame: frame }, { transfer: frame });
and you expect only histogram to be transferred (since that's all that
supports it when you write this code), your code breaks when
CanvasPixelArray later supports it.
b) Transfer lists + separate transferMessage method. We still equip
postMessage with transfer lists, these transfer lists list ArrayBuffers, and
we provide a separate method transferMessage with recursive transfer
semantics.
What do people think?
Same problem.
If you want a quicker way to transfer all messages of given types, see
my previous mail: { transfer: ArrayBuffer }.
Agreed on these points. Using an object graph for the transfer list
(which is what the recursive transfer list idea boils down to) also
sounds overly complicated.
May I suggest to reconsider adding another optional array argument to
postMessage for the transfer list, rather than using an object with
special properties?
Points in favor of adding another optional array argument:
1. Less typing, and less possibility that a typo will cause incorrect behavior:
worker.postMessage(objectGraph, null, [ arrayBuffer1ToTransfer,
arrayBuffer2ToTransfer ]);
vs.
worker.postMessage(objectGraph, { transfer: [
arrayBuffer1ToTransfer, arrayBuffer2ToTransfer] });
2. Possibility of using Web IDL to specify the type of the optional
array argument (i.e., Transferable[]?). Would be harder to do using
an object -- requiring either specifying another interface type with
the ports and transfer attributes, or using any plus
ECMAScript-specific text.
Points in favor of using an object:
1. Could potentially overload the meaning of the optional ports array
argument, avoiding adding another argument to postMessage.
2. More extensible in the future.
Thoughts?
-Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Fri, Jun 3, 2011 at 2:15 PM, Andrew Wilson atwil...@google.com wrote:
On Fri, Jun 3, 2011 at 1:02 PM, Jonas Sicking jo...@sicking.cc wrote:
On Fri, Jun 3, 2011 at 11:37 AM, Kenneth Russell k...@google.com wrote:
On Fri, Jun 3, 2011 at 9:46 AM, Glenn Maynard gl...@zewt.org wrote:
On Fri, Jun 3, 2011 at 11:12 AM, Dmitry Lomov dslo...@google.com
wrote:
a) Recursive transfer lists. Allow arbitrary objects, not only
ArrayBuffers,
to appear in transfer lists. ArrayBuffers that are under objects in
transfer lists are transferred, others are cloned.
This again causes the same forwards-compatibility problem. If you do
this:
frame = { video: canvasPixelArrayInstance, histogram: arrayBuffer } }
postMessage({ frame: frame }, { transfer: frame });
and you expect only histogram to be transferred (since that's all that
supports it when you write this code), your code breaks when
CanvasPixelArray later supports it.
b) Transfer lists + separate transferMessage method. We still equip
postMessage with transfer lists, these transfer lists list
ArrayBuffers, and
we provide a separate method transferMessage with recursive transfer
semantics.
What do people think?
Same problem.
If you want a quicker way to transfer all messages of given types, see
my previous mail: { transfer: ArrayBuffer }.
Agreed on these points. Using an object graph for the transfer list
(which is what the recursive transfer list idea boils down to) also
sounds overly complicated.
May I suggest to reconsider adding another optional array argument to
postMessage for the transfer list, rather than using an object with
special properties?
Points in favor of adding another optional array argument:
1. Less typing, and less possibility that a typo will cause incorrect
behavior:
worker.postMessage(objectGraph, null, [ arrayBuffer1ToTransfer,
arrayBuffer2ToTransfer ]);
vs.
worker.postMessage(objectGraph, { transfer: [
arrayBuffer1ToTransfer, arrayBuffer2ToTransfer] });
2. Possibility of using Web IDL to specify the type of the optional
array argument (i.e., Transferable[]?). Would be harder to do using
an object -- requiring either specifying another interface type with
the ports and transfer attributes, or using any plus
ECMAScript-specific text.
Points in favor of using an object:
1. Could potentially overload the meaning of the optional ports array
argument, avoiding adding another argument to postMessage.
2. More extensible in the future.
Thoughts?
My first thought is that so far no implementer has stepped up and said
that changing the meaning of the 'ports' argument would not be
acceptable. Would be great if someone who is reading this thread and
who works at Google/Apple/Opera could check with the relevant people
to see if such a change would be possible.
It's certainly possible - there's nothing intrinsically difficult with
making this change from an implementor's point of view (I've had my fingers
in both the WebKit and Chromium MessagePort implementations so I'm
relatively confident that this would not be prohibitively hard).
Is it desirable? My knee-jerk reaction is that we should stick with changes
that are compatible with the existing API (like Ian's original suggestion,
or adding a separate optional array of transferable objects) - I have no
data on the number of sites that are using the current API but I don't think
we should be changing existing APIs with multiple implementations without
significant motivation. The stated motivations for breaking this API don't
seem compelling to me given the existence of backwards-compatible
alternatives.
Drew pointed me off-list to the original discussion motivating the
addition of the optional argument for the array of ports:
http://lists.w3.org/Archives/Public/public-html-comments/2009Mar/0001.html
As I've been thinking more about Ian's original proposal, it seems to
me that I might have misunderstood the intent.
The current optional array of MessagePorts is very close to supporting
the semantic of transfer of ownership that is desired. If a
MessagePort is passed in this array, its ownership is transferred to
the other side. The undesirable aspects seem to be:
1) It doesn't seem to be allowed to refer to those MessagePort objects
in the object graph which is the first argument to postMessage. At
least, it looks to me like the structured clone algorithm will throw a
DATA_CLONE_ERR exception if it encounters a MessagePort object.
2) The array of ports shows up on the other side as the ports
property of the MessageEvent.
(1) could be solved, and in a backward-compatible way, by defining the
behavior of MessagePort under structured clone. Cloning a
MessagePort would cause a DATA_CLONE_ERR exception to be thrown, but
if structured clone were taught about the objects to be transferred,
then if a MessagePort were transferred, the cloned object graph could
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Fri, Jun 3, 2011 at 4:15 PM, Andrew Wilson atwil...@google.com wrote:
On Fri, Jun 3, 2011 at 3:23 PM, Glenn Maynard gl...@zewt.org wrote:
On Fri, Jun 3, 2011 at 5:15 PM, Andrew Wilson atwil...@google.com wrote:
significant motivation. The stated motivations for breaking this API
don't
seem compelling to me given the existence of backwards-compatible
alternatives.
This proposal is backwards-compatible. If the argument is an array,
nothing changes, so postMessage(..., [ports]) is equivalent to
postMessage(..., {ports: [ports]}). (The array-only approach can be
done compatibly, too; the object version is just an alternative to
that.) What's backwards-incompatible?
Ah, I missed that piece (to be honest, I haven't been following this
discussion in every detail - I only chimed in because of Jonas' request for
implementation feedback).
For anyone not looking closely at the IDL while reading this, this
means deprecating (for whatever value deprecate has on the web) the
ports array in MessageEvent--not the ports parameter to postMessage
(that's a sequence).
Does this affect the API for the SharedWorker onconnect message as well?
Good point; that might inform not deprecating the ports array in
MessageEvent, but leaving it as is.
-Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
(It would have been better not to fork the thread with a different
subject line...)
On Thu, Jun 2, 2011 at 9:58 AM, Travis Leithead
travis.leith...@microsoft.com wrote:
I'm a little concerned about the inherit approach that Ian outlines...
This plan requires all objects that want to opt-in to a new
transfer-of-ownership (or really any special custom behavior for postMessage)
to 1) participate in the special inheritance interface and 2) be isolated
from the primary object graph being passed to the first parameter. For
example, instead of allowing the structured clone algorithm to preserve all
my object relationships as-is, I will have to detach some objects from the
graph (to include in the proposed 2nd parameter to postMessage) and then
re-attach them on the destination side.
I realize I misunderstood the proposal. I thought the additional array
of objects was supposed to indicate those in the object graph that
wanted ownership to be transferred. Having to send these objects
separately from the rest of those in the graph is undesirable and in
my opinion unworkable.
Isn't there some way we can flag the objects that want transfer of ownership
by a property or identifier on the objects themselves? That way, they won't
have to be special-cased into a separate list. For example, if the
ArrayBuffer added a property indicating transfer-of-ownership? Or a new
configuration option for postMessage ( { transferOwnership: true } )?
A per-object property is undesirable because it forces all types that
want to participate in transfer of ownership to add a new property
specifically for the Web Messaging spec.
I still like the idea of identifying those objects in the graph passed
to postMessage which should be transferred rather than copied, since
it's flexible and should be easy to use. (Though maybe not, if the
object graph is deep and it's difficult to pull out all the objects
you want to transfer?) Adding a flag to postMessage, or adding a new
entry point like transferMessage, also seem fine.
-Ken
-Original Message-
From: Arthur Barstow [mailto:art.bars...@nokia.com]
Sent: Thursday, June 02, 2011 9:02 AM
To: ext Jonas Sicking; Kenneth Russell; Ian Hickson
Cc: Travis Leithead; g...@google.com; cmar...@apple.com; gl...@zewt.org;
public-webapps@w3.org
Subject: What changes to Web Messaging spec are proposed? [Was: Re: Using
ArrayBuffer as payload for binary data to/from Web Workers]
What are the specific change(s) to the Web Messaging spec being proposed:
http://dev.w3.org/html5/postmsg/
-AB
On Jun/2/2011 11:25 AM, ext Jonas Sicking wrote:
On Wed, Jun 1, 2011 at 4:55 PM, Kenneth Russellk...@google.com wrote:
On Tue, May 31, 2011 at 3:35 PM, Ian Hicksoni...@hixie.ch wrote:
On Tue, 31 May 2011, Kenneth Russell wrote:
Jonas's suggestion of adding another argument to postMessage, and
Gregg's generalization to declare it as an array of objects to be
transferred rather than copied, sounds good.
We could change make MessagePort and ArrayBuffer both inherit from a
[NoInterfaceObject] empty interface, and then make the MessagePort[]
argument of the various postMessage() methods instead take an array
of this new interface, and then just have ArrayBuffer and
MessagePort both define how to be cloned in this way.
If people like this approach I can work with Kenneth on getting the
wording right in the various specs.
This sounds good to me; in the interest of moving things forward, are
there any objections?
No, this sounded good to the people here at mozilla that I talked with
about this.
/ Jonas
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Thu, Jun 2, 2011 at 8:25 AM, Jonas Sicking jo...@sicking.cc wrote: On Wed, Jun 1, 2011 at 4:55 PM, Kenneth Russell k...@google.com wrote: On Tue, May 31, 2011 at 3:35 PM, Ian Hickson i...@hixie.ch wrote: On Tue, 31 May 2011, Kenneth Russell wrote: Jonas's suggestion of adding another argument to postMessage, and Gregg's generalization to declare it as an array of objects to be transferred rather than copied, sounds good. We could change make MessagePort and ArrayBuffer both inherit from a [NoInterfaceObject] empty interface, and then make the MessagePort[] argument of the various postMessage() methods instead take an array of this new interface, and then just have ArrayBuffer and MessagePort both define how to be cloned in this way. If people like this approach I can work with Kenneth on getting the wording right in the various specs. This sounds good to me; in the interest of moving things forward, are there any objections? No, this sounded good to the people here at mozilla that I talked with about this. Apologies but I misunderstood something about this proposal and it no longer seems desirable. I've followed up on the other (forked) thread. -Ken
Re: What changes to Web Messaging spec are proposed? [Was: Re: Using ArrayBuffer as payload for binary data to/from Web Workers]
On Thu, Jun 2, 2011 at 12:53 PM, David Levin le...@chromium.org wrote: In summary, there is a desire for a mechanism to transfer objects (to allow for potentially better perf) across a MessagePort. The mechanism: needs to have an intuitive feel for developers, must preserve backwards compatibility, should ideally allow the port to function the same regardless of whether the message was cloned or transferred. should be easy to use. There are three ideas for how to accomplish this: 1. Mixing in the list of objects to be cloned with the ports and use that list to determine what objects in the message should be cloned. This allows a lot of flexibility. It feels odd mixing in a list of objects with the ports when the two have nothing related. It also feels complicated having to add objects in two places (the message and this extra array). Another option which was mentioned: 1a) Add another optional argument to postMessage after the array of ports, which would be an array of objects to transfer rather than clone. I think 1a) is the best way to expose the functionality. As Glenn pointed out, for backward and forward compatibility reasons, it's best if the developer explicitly selects the objects to transfer. The desire would be for this change to apply not just to the postMessage method on MessagePort and Worker but also to that on Window. -Ken 2. Adding another parameter to postMessage clone/transfer or true/false, etc. It is less flexible than 1. It is very simple and easy to use. It may not be as noticeable when reading the code that this postMessage does a transfer of items. 3. Adding another method transferMessage with the same parameters as postMessage. It is less flexible than 1. It is very simple and easy to use. It may be a pain to keep this in sync with postMessage. It should be very noticeable when reading code. What do you think is the best way to expose this to web developers? dave
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Tue, May 31, 2011 at 3:35 PM, Ian Hickson i...@hixie.ch wrote: On Tue, 31 May 2011, Kenneth Russell wrote: Jonas's suggestion of adding another argument to postMessage, and Gregg's generalization to declare it as an array of objects to be transferred rather than copied, sounds good. We could change make MessagePort and ArrayBuffer both inherit from a [NoInterfaceObject] empty interface, and then make the MessagePort[] argument of the various postMessage() methods instead take an array of this new interface, and then just have ArrayBuffer and MessagePort both define how to be cloned in this way. If people like this approach I can work with Kenneth on getting the wording right in the various specs. This sounds good to me; in the interest of moving things forward, are there any objections? -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Tue, May 31, 2011 at 11:33 AM, Travis Leithead travis.leith...@microsoft.com wrote: The editors' draft of the typed array spec has been updated with a strawman proposal for this zero-copy, transfer-of-ownership behavior: http://www.khronos.org/registry/typedarray/specs/latest/ Feedback would be greatly appreciated. For the purposes of keeping the conversation centralized, it might be helpful if we could use the public_webgl list; see https://www.khronos.org/webgl/public-mailing-list/ . While I see the need for this, i think it will be very surprising to authors that for all other data, postMessage is purely a read-only action. However for ArrayBuffers it would not be. There are two ways we can improve this situation: 1. Add a separate method next to postMessage which has the prescribed functionality. This also has the advantage that it lets users choose if they want the transfer-ownership functionality or not, for example for cases when performance isn't as big requirement, and when ArrayBuffers are small enough that the transferring ownership logic adds more overhead than memory copying logic would. 2. Add a separate argument to postMessage, similar to the 'ports' argument, which contains a list of array buffers whose ownership should be transferred. Riffing off idea #2, the second argument could be an array of objects who's ownership should be transferred. For now only ArrayBuffers would be legal objects but at some point in the future other types of objects could be added (not sure what those objects would be but that's a much more flexible interface than #1. You can chose to copy some ArrayBuffers and transfer others. I tend to agree with Jonas on this one-having an ArrayBuffer stop working on either the primary document or a web worker after posting it seems like a bad developer experience by default. Having an opt-in transfer of ownership seems like a better idea, though I don't like special-casing ArrayBuffers, as I'd probably want to do this for large ImageData objects as well (with their associated CanvasPixelArrays). After discussing this a bit internally, we raised four major arguments against default transfer of ownership for TypedArrays: 1. User complexity: Transfer of ownership is more complicated for developers, and does not fit with the silent and unobtrusive model of cloning that is typical of SCA. An operation that makes the object unusable should be explicit, not implicit. 2. SCA behavior for TypedArrays should align with Blob, etc.: Using transfer of ownership for one and not the other will lead to user confusion. Generally, SCA has not embraced transfer of ownership as the user model, and we don't believe TypedArrays should default to this very different clone behaviour. 3. Cross-thread assumption: The idea of transfer of ownership semantics strongly suggests usage of cloning across threads. SCA is used in many other scenarios where transfer of ownership does not add value, but does hurt usability. For example, Workers implemented to run in another process (a Web Worker implementation detail), IndexedDb (long-term storage of a SCA object graph), etc. In the case of IndexedDb for example, transfer of ownership has an unexpected semantic, as the database itself doesn't have a notion of ownership. 4. Split definitions of SCA algorithm: Having the specification of SCA behaviour for TypedArrays be separate than the definition in the HTML5 spec is likely to lead to continued divergence of the SCA algorithm. It would be better to define this in one spec (i.e., in HTML5) I agree that it would be better to generalize the transfer of ownership mechanism to support more types in the future. The original motivation for making transfer of ownership the behavior for Typed Arrays under structured clone and/or postMessage was solely to minimize changes to the HTML spec; it would be better to come up with a more general solution. Jonas's suggestion of adding another argument to postMessage, and Gregg's generalization to declare it as an array of objects to be transferred rather than copied, sounds good. Adding a transferMessage API doesn't sound as good since it will require larger code changes to take advantage of it, and is less flexible. I'll investigate updating the typed array proposals in this direction. Transfer of ownership of buffers is valuable even when Workers are implemented in another process. The first time an ArrayBuffer is posted from a Worker to the primary document, its storage could be promoted (as an implementation detail) to shared memory. Once the document is done with the data, it would post the buffer back to the worker for re-filling. Subsequent ping-ponging would not involve any further data copies. This is the primary goal of these typed array spec updates: to enable efficient producer-consumer queues between workers and the document. Note
Re: [XHR2] responseType and response properties
On Sat, May 21, 2011 at 12:36 AM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Firefox 6 is going to add support for the the new responseType and response properties. We would have liked to release these as moz-prefixed properties, but it appears that webkit has already shipped them unprefixed, thus there's not much point in prefixing. Both because we likely can't fix bugs in them anyway, and because sites have already started depending on them. I'd really like to reiterate that we need to be careful about releasing newly minted APIs unprefixed. Even in the case where we here on the list agree on the desired behavior. The Blob.slice mess (which we fortunately managed to save last minute) is a good example of why we should give API time to mature before unprefixing. So in that spirit, I have a questions about the actual behavior of XHR.response: First, should it return a new ArrayBuffer every time the property is accessed (This is how the mozResponseArrayBuffer property in FF4 behaves), or should it return the same buffer every time? Since arraybuffers are mutable, there's a risk that one consumer will modify the response such that all other consumers see a modified result. On the other hand creating a new buffer every time can be quite expensive (at least without complex/slow copy-on-write implementations). It also would result in the surprising property that xhr.response != xhr.response. In FF6 we're aiming to return the same buffer every time. The other-consumers-will-see-a-modified-result behavior is unfortunate, but similar to how responseXML behaves and hasn't seemed to cause a problem there. I think there was some discussion on this list a while ago about the semantics, and that the consensus at the time was that a copy was needed to avoid the mutation problem you mention. I agree that the copy is problematic and expensive. Some changes to the Typed Array spec are in progress, one of which is the addition of a read-only ArrayBuffer. Once this is finalized in the Typed Array spec and implemented in browsers, XHR could return a read-only ArrayBuffer from .response. Any comments about this would be welcome. The discussion has mostly been going on on the WebGL mailing list: https://www.khronos.org/webgl/public-mailing-list/ . -Ken Second, what should .response return before the download finishes? It would be nice if you could access partial results. However ArrayBuffers are fixed size and we won't know the total size until the whole download is finished. We could return a copy on each access, but again that would be somewhat expensive. For now we're going to return null during loading in FF6 when .responseType is set to arraybuffer. / Jonas
Re: [XHR2] responseType and response properties
On Mon, May 23, 2011 at 4:27 PM, Jonas Sicking jo...@sicking.cc wrote: On Mon, May 23, 2011 at 12:27 PM, Kenneth Russell k...@google.com wrote: On Sat, May 21, 2011 at 12:36 AM, Jonas Sicking jo...@sicking.cc wrote: Hi All, Firefox 6 is going to add support for the the new responseType and response properties. We would have liked to release these as moz-prefixed properties, but it appears that webkit has already shipped them unprefixed, thus there's not much point in prefixing. Both because we likely can't fix bugs in them anyway, and because sites have already started depending on them. I'd really like to reiterate that we need to be careful about releasing newly minted APIs unprefixed. Even in the case where we here on the list agree on the desired behavior. The Blob.slice mess (which we fortunately managed to save last minute) is a good example of why we should give API time to mature before unprefixing. So in that spirit, I have a questions about the actual behavior of XHR.response: First, should it return a new ArrayBuffer every time the property is accessed (This is how the mozResponseArrayBuffer property in FF4 behaves), or should it return the same buffer every time? Since arraybuffers are mutable, there's a risk that one consumer will modify the response such that all other consumers see a modified result. On the other hand creating a new buffer every time can be quite expensive (at least without complex/slow copy-on-write implementations). It also would result in the surprising property that xhr.response != xhr.response. In FF6 we're aiming to return the same buffer every time. The other-consumers-will-see-a-modified-result behavior is unfortunate, but similar to how responseXML behaves and hasn't seemed to cause a problem there. I think there was some discussion on this list a while ago about the semantics, and that the consensus at the time was that a copy was needed to avoid the mutation problem you mention. I agree that the copy is problematic and expensive. Some changes to the Typed Array spec are in progress, one of which is the addition of a read-only ArrayBuffer. Once this is finalized in the Typed Array spec and implemented in browsers, XHR could return a read-only ArrayBuffer from .response. Any comments about this would be welcome. The discussion has mostly been going on on the WebGL mailing list: https://www.khronos.org/webgl/public-mailing-list/ . I'm not actually sure that returning a read-only ArrayBuffer is always advantageous. In the by far most common case, there is only one consumer of a specific XHR request. In such a situation modifying the response does not run the risk of confusing other consumers as there are none. So in that situation it would be wasteful to force the consumer to copy the data just to be able to modify it. The question to ask is whether it's common for that consumer to modify the result of the XHR. In all of the use cases I'm aware of, it's usually processed (maybe manually decompressed), sliced up and/or passed to some other API, but not actually modified in place. Additionally, returning a new ArrayBuffer, even if read-only, still creates the confusing situation where xhr.response != xhr.response. If the ArrayBuffer were read-only then it would not be necessary to return a new one each time .response were called. Finally, since webkit already has release XHR.response unprefixed, it's going to be hard to change its behavior, especially from read/write to read-only. That's certainly possible. I don't know how widely used the ArrayBuffer response type is or what the compatibility impact would be. However, I've received some anecdotal feedback from colleagues that they would welcome efficiency improvements in this area (assuming that the current ArrayBuffer response were copied each time it was fetched). However adding a readonly ArrayBuffer would allows us to do things like xhr.responseType = readonly-arraybuffer if we want to. Agreed. -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 6:17 PM, Kenneth Russell k...@google.com wrote: On Mon, Mar 7, 2011 at 5:18 PM, Chris Marrin cmar...@apple.com wrote: On Mar 7, 2011, at 4:46 PM, Kenneth Russell wrote: On Mon, Mar 7, 2011 at 3:54 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 6:05 PM, Chris Marrin cmar...@apple.com wrote: Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. ArrayBuffer is the most obvious use for zero-copy messaging, but I don't think it should be limited to it... Has there been discussion anywhere that I've missed? Probably not the only one, but check the WebWorkers and images thread on whatwg. There's definitely interest among the editors of the Typed Array spec in revising the spec to support zero-copy data transfers to and from web workers. In informal offline discussions, there was a tentative plan to put up a new draft for discussion within the next month or so. A goal was to prototype it before solidifying a spec so that we can be assured it will work well for real-world use cases. Yeah, I guess the question is whether we should put the functionality into ArrayBuffer, or into a wrapper class which would part of the Web Worker spec. The latter might make it easier to add other resources (like image and canvas) at some point. But I agree, it should be implemented before finalizing anything. Did I hear you volunteer to add a strawman proposal to the Typed Array spec? :-) Yes, you did. :-) The editors' draft of the typed array spec has been updated with a strawman proposal for this zero-copy, transfer-of-ownership behavior: http://www.khronos.org/registry/typedarray/specs/latest/ Feedback would be greatly appreciated. For the purposes of keeping the conversation centralized, it might be helpful if we could use the public_webgl list; see https://www.khronos.org/webgl/public-mailing-list/ . Thanks, -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 5:55 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 8:04 PM, Chris Marrin cmar...@apple.com wrote: Probably not the only one, but check the WebWorkers and images thread on whatwg. Yeah, I thought about that case. The extra complication there is that images are rendering resources, which muddies the issues some. If the 2D Canvas API were able to use ArrayBuffer data as an image source like WebGL can, then we could kill 2 birds with one stone. Pass the ArrayBuffer to the worker and have it generate some image data, then pass it back and render it to a canvas. You could even split the array buffer into tiles and have multiple workers operate on the tiles. I'd expect CanvasPixelArray to allow optimizations that ArrayBuffer doesn't, since the implementation can use the native surface format, translating to RGBA for the script transparently. This can matter for streaming textures to OpenGL/D3D, too; creating BGRA textures on nVidia hardware is typically much faster than RGBA ones. I don't recall if this has been brought up: are there cases where explicit zero-copy messaging is better than transparent copy-on-write? Yes. Copy on write does not efficiently handle the case where large amounts of data are continually produced by workers and posted to the main thread for display. Each time the worker posts a block of data to the main thread, the next time it attempts to update its version of the block for the next iteration, a copy will need to be made so the main thread's version appears immutable. Bi-directional, zero-copy messaging is needed to preserve ECMAScript's shared-nothing semantic while avoiding a continuous stream of memory allocation for producer/consumer queues between workers and the main thread. The Vertex Buffer Object demo at http://khronos.org/webgl/wiki/Demo_Repository provides an example where multiple workers should be able to produce independent portions of the mesh for rendering by the main thread, and where a nearly linear speedup is possible. -Ken
Re: Using ArrayBuffer as payload for binary data to/from Web Workers
On Mon, Mar 7, 2011 at 5:18 PM, Chris Marrin cmar...@apple.com wrote: On Mar 7, 2011, at 4:46 PM, Kenneth Russell wrote: On Mon, Mar 7, 2011 at 3:54 PM, Glenn Maynard gl...@zewt.org wrote: On Mon, Mar 7, 2011 at 6:05 PM, Chris Marrin cmar...@apple.com wrote: Now that ArrayBuffer has made its way into XHR, I think it would be reasonable to somehow use this new object type as a way to pass data to and from Workers without copying. I've seen hints and thoughts about this here and there, but I've never seen a formal discussion. I'm not even sure if webapps is the place for this discussion, although it seems like a reasonable place. Please let me know if there is a better place. ArrayBuffer is the most obvious use for zero-copy messaging, but I don't think it should be limited to it... Has there been discussion anywhere that I've missed? Probably not the only one, but check the WebWorkers and images thread on whatwg. There's definitely interest among the editors of the Typed Array spec in revising the spec to support zero-copy data transfers to and from web workers. In informal offline discussions, there was a tentative plan to put up a new draft for discussion within the next month or so. A goal was to prototype it before solidifying a spec so that we can be assured it will work well for real-world use cases. Yeah, I guess the question is whether we should put the functionality into ArrayBuffer, or into a wrapper class which would part of the Web Worker spec. The latter might make it easier to add other resources (like image and canvas) at some point. But I agree, it should be implemented before finalizing anything. Did I hear you volunteer to add a strawman proposal to the Typed Array spec? :-) Yes, you did. :-) -Ken
Re: Mouse Capture for Canvas
This API doesn't handle all of the desired use cases. In particular, to implement Quake-style mouse look (needed for e.g. http://code.google.com/p/quake2-gwt-port/) it needs to work when the mouse button is up, not just down. -Ken On Tue, Feb 8, 2011 at 12:11 PM, Robert O'Callahan rob...@ocallahan.org wrote: IE has a setCapture DOM API. We've implemented it in Gecko: https://developer.mozilla.org/en/DOM/element.setCapture Rob -- Now the Bereans were of more noble character than the Thessalonians, for they received the message with great eagerness and examined the Scriptures every day to see if what Paul said was true. [Acts 17:11]
Re: Mouse Capture for Canvas
On Tue, Feb 8, 2011 at 5:49 PM, Robert O'Callahan rob...@ocallahan.org wrote: On Wed, Feb 9, 2011 at 11:37 AM, Kenneth Russell k...@google.com wrote: This API doesn't handle all of the desired use cases. In particular, to implement Quake-style mouse look (needed for e.g. http://code.google.com/p/quake2-gwt-port/) it needs to work when the mouse button is up, not just down. How would you satisfy that use-case without enabling abusive behavior by Web sites? Per the proposal posted earlier and at http://www.w3.org/Bugs/Public/show_bug.cgi?id=9557 : 1. Only enable mouse capture in response to a user gesture (clicking on the element). 2. Possibly require confirmation from the user to enter mouse capture mode. 3. Present clear UI to indicate how to exit mouse capture mode. -Ken (I think that case would work with our API if the game was full-screen, which I expect it normally would be.) Rob -- Now the Bereans were of more noble character than the Thessalonians, for they received the message with great eagerness and examined the Scriptures every day to see if what Paul said was true. [Acts 17:11]
Re: [XHR2] ArrayBuffer integration
On Thu, Sep 23, 2010 at 2:42 AM, Anne van Kesteren ann...@opera.com wrote: On Wed, 08 Sep 2010 19:55:33 +0200, Kenneth Russell k...@google.com wrote: Mozilla's experimental name is mozResponseArrayBuffer, so perhaps to avoid collisions the spec could call it responseArrayBuffer. While I do not think there would be collision (at least not in ECMAScript, which is what we are designing for) naming it responseArrayBuffer is fine with me. And also now done that way in the draft. Still need to get a saner reference to the ArrayBuffer specification than https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html though. :-) http://dev.w3.org/2006/webapi/XMLHttpRequest-2/ Thanks, this is great and very exciting. This motivates implementing the proposed DataView interface ( https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html#6 ), which will make it easier to read multi-byte values with specified endianness out of an ArrayBuffer. For WebKit I've filed https://bugs.webkit.org/show_bug.cgi?id=46541 . -Ken (You can also do send(ArrayBuffer) obviously. I personally think supporting this for both BlobBuilder and send() makes sense. That way Blob/File etc. work too.) -- Anne van Kesteren http://annevankesteren.nl/
Re: [XHR2] ArrayBuffer integration
On Fri, Sep 24, 2010 at 5:36 PM, Jian Li jia...@chromium.org wrote: I plan to add ArrayBuffer support to BlobBuilder and FileReader. Chris, it is good that you would pick up the work for XHR. We can talk about how we're going to add ArrayBufferView to read ArrayBuffer. All of the Typed Array view types (Uint8Array, Float32Array, etc.) except for Float64Array are already implemented in WebKit. The major missing one for file and network I/O is DataView. -Ken Jian On Fri, Sep 24, 2010 at 5:23 PM, Kenneth Russell k...@google.com wrote: On Thu, Sep 23, 2010 at 2:42 AM, Anne van Kesteren ann...@opera.com wrote: On Wed, 08 Sep 2010 19:55:33 +0200, Kenneth Russell k...@google.com wrote: Mozilla's experimental name is mozResponseArrayBuffer, so perhaps to avoid collisions the spec could call it responseArrayBuffer. While I do not think there would be collision (at least not in ECMAScript, which is what we are designing for) naming it responseArrayBuffer is fine with me. And also now done that way in the draft. Still need to get a saner reference to the ArrayBuffer specification than https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html though. :-) http://dev.w3.org/2006/webapi/XMLHttpRequest-2/ Thanks, this is great and very exciting. This motivates implementing the proposed DataView interface ( https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html#6 ), which will make it easier to read multi-byte values with specified endianness out of an ArrayBuffer. For WebKit I've filed https://bugs.webkit.org/show_bug.cgi?id=46541 . -Ken (You can also do send(ArrayBuffer) obviously. I personally think supporting this for both BlobBuilder and send() makes sense. That way Blob/File etc. work too.) -- Anne van Kesteren http://annevankesteren.nl/
Re: [XHR2] ArrayBuffer integration
On Wed, Sep 8, 2010 at 8:44 AM, Julian Reschke julian.resc...@gmx.de wrote: On 08.09.2010 17:35, Anne van Kesteren wrote: ... Okay. I guess we can all add support for it and see who screams :-) I can certainly add this to XMLHttpRequest Level 2 and have been wanting to do that since forever. As you might have seen notes to that effect are in the specification. send() can be overloaded as per usual. Currently the proposal for the response member is responseBody which nicely maps to HTTP entity body. ... It does, but doesn't it clash with responseBody in Microsoft's ActiveX control? (http://msdn.microsoft.com/en-us/library/ms534368%28v=VS.85%29.aspx) Mozilla's experimental name is mozResponseArrayBuffer, so perhaps to avoid collisions the spec could call it responseArrayBuffer. -Ken
Re: [whatwg] ArrayBuffer and ByteArray questions
On Wed, Sep 8, 2010 at 11:21 AM, Oliver Hunt oli...@apple.com wrote: On Sep 8, 2010, at 11:13 AM, Chris Marrin wrote: Web Sockets is certainly another candidate, but I meant Web Workers. There have been informal discussions on using ArrayBuffers as a way to safely share binary data between threads. I don't believe anything has been formalized here. You can't share data between workers. There is no (and there cannot be) any shared state between multiple threads of JS execution. Let's say efficiently send rather than share. The current thinking has been around a way to post one ArrayBuffer to a worker which would close that ArrayBuffer and all views on the main thread. The way to get the same backing store from the worker back to the main thread would be to post the ArrayBuffer from the worker to the main thread, at which point the ArrayBuffer and all views on the worker would be closed. This ping-ponging would allow efficient implementation of producer/consumer queues without allocating new backing store each time the worker wants to produce something for the main thread. This would require some small API additions to the typed array spec, and a prototype so we can convince ourselves of its effectiveness. -Ken
Re: [whatwg] ArrayBuffer and ByteArray questions
On Wed, Sep 8, 2010 at 5:04 PM, Silvia Pfeiffer silviapfeiff...@gmail.com wrote: On Thu, Sep 9, 2010 at 4:37 AM, Chris Marrin cmar...@apple.com wrote: On Sep 8, 2010, at 11:21 AM, Oliver Hunt wrote: On Sep 8, 2010, at 11:13 AM, Chris Marrin wrote: Web Sockets is certainly another candidate, but I meant Web Workers. There have been informal discussions on using ArrayBuffers as a way to safely share binary data between threads. I don't believe anything has been formalized here. You can't share data between workers. There is no (and there cannot be) any shared state between multiple threads of JS execution. Right. I didn't mean literal sharing. But you can imagine some copy-on-write semantics which would make it more efficient to pass data this way. Is this then similar to posting ImageData with Web Workers? (http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#imagedata) . I know that these can already be put into a postMessage and they are effectively arrays. It's similar, but we want to define different semantics to achieve higher performance. Copy-on-write does not work because in a producer/consumer scenario the producer will always overwrite the same buffer passed to the consumer, leading to a copy each time. We want to make the source ArrayBuffer, and any ArrayBufferViews, zero length upon posting them to a worker or back to the main thread. By ping-ponging the same ArrayBuffer back and forth you can avoid allocating new backing store each iteration. -Ken
Re: ArrayBuffer and ByteArray questions
On Tue, Sep 7, 2010 at 4:19 PM, Nathan nat...@webr3.org wrote: Jian Li wrote: Hi, Several specs, like File API and WebGL, use ArrayBuffer, while other spec, like XMLHttpRequest Level 2, use ByteArray. Should we change to use the same name all across our specs? Since we define ArrayBuffer in the Typed Arrays spec ( https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html), should we favor ArrayBuffer? In addition, can we consider adding ArrayBuffer support to BlobBuilder, FormData, and XMLHttpRequest.send()? which reminds me, I meant to ask if the aforementioned TypedArray spec should be brought in to webapps / w3c land? seems to complement the other base types used in webidl etc rather well + my gut reaction was why isn't this standardized within w3c? There's no particular reason why the Typed Array spec is being standardized in the Khronos group, aside from the fact that these array-like types originated in the WebGL spec and have evolved to meet use cases specified by WebGL. We have been hoping that they would have uses outside of WebGL, and some discussions have occurred with working groups such as TC39 to see how they might be better specified and standardized. We'd be open to hosting the spec development elsewhere. Vlad mentioned to me off-list that Mozilla has implemented an experimental mozResponseArrayBuffer on XHR objects, and will likely do the same on the File API to add a readAsArrayBuffer alongside readAsBinaryString etc. -Ken
Re: [whatwg] An BinaryArchive API for HTML5?
On Thu, Jul 30, 2009 at 6:13 AM, Sebastian
Markbågesebast...@calyptus.eu wrote:
This suggestion seems similar to Digg's Stream project that uses multipart
documents: http://github.com/digg/stream
While it would be nice to have a way to parse and handle this in JavaScript,
it shouldn't be JavaScript's responsibility to work with large object data
and duplicating it as in-memory data strings.
The real issue here is the overhead of each additional HTTP request for
those thousands of objects. But that's useful for all parts of the spec if
you can download it as a single package even without JavaScript. Images,
CSS, background-images, JavaScript, etc. Currently you can include graphics
as data URLs in CSS. Using a package you could package whole widgets (or
apps) as a single request.
I'd suggest that this belongs in a lower level API such as the URIs and
network stack for the tags. You could specify a file within an archive by
adding an hash with the filename to the URI:
img src=http://someplace.com/somearchive.tgz#myimage.jpg; /
style type=text/css
#id { background-image:
url(http://someplace.com/somearchive.tgz#mybackgroundimage.jpg); }
/style
script src=http://someplace.com/somearchive.tgz#myscript.js;
type=text/javascript/script
var img = new Image();
img.src = http://someplace.com/somearchive.tgz#myimage.png;;
Now which packaging format to use would be a discussion on it's own. An easy
route would be to use multipart/mixed that is already used for this in
e-mails and can also be gzipped using Content-Encoding.
In the context of the 3d canvas discussions, it looks like there is a
need to load binary blobs of vertex data and feed them to the graphics
card via a JavaScript call. Here is some hypothetical IDL similar to
what is being considered:
[IndexGetter, IndexSetter]
interface CanvasFloatArray {
readonly attribute unsigned long length;
};
interface CanvasRenderingContextGL {
...
typedef unsigned long GLenum;
void glBufferData(in GLenum target, in CanvasFloatArray data,
in GLenum usage);
...
};
Do you have some suggestions for how the data could be transferred
most efficiently to the glBufferData call? As far as I know there is
no tag which could be used to refer to the binary file within the
archive. If there were then presumably it could provide its contents
as a CanvasFloatArray or other type.
-Ken
On Thu, Jul 30, 2009 at 11:41 AM, Anne van Kesteren ann...@opera.com
wrote:
On Thu, 30 Jul 2009 08:49:12 +0200, Gregg Tavares g...@google.com wrote:
What are people's feelings on adding a Binary Archive API to HTML5?
I think it makes more sense to build functionality like this on top of the
File API rather than add more things into HTML5.
It seems like it would be useful if there was browser API that let you
download something like gzipped tar files.
We already have that: XMLHttpRequest.
The API would look something like
var request = createArchiveRequest();
request.open(GET, http://someplace.com/somearchive.tgz;);
request.onfileavailable = doSomethingWithEachFileAsItArrives;
request.send();
I don't think we should introduce a new HTTP API.
function doSomethingWithEachFileAsItArrives(binaryBlob) {
// Load every image in archive
if (binaryBlob.url.substr(-3) == .jpg) {
var image = new Image();
image.src = binaryBlob.toDataURL(); // or something;
...
}
// Look for a specific text file
else if (binaryBlog.url === myspecial.txt) {
// getText only works if binaryBlob is valid utf-8 text.
var text = binaryBlob.getText();
document.getElementById(content).innerHTML = text;
}
}
Having dedicated support for a subset of archiving formats in within the
API for File objects makes sense to me. Latest draft of the File API I know
of is
http://dev.w3.org/2006/webapi/FileUpload/publish/FileAPI.xhtml
and the mailing list would be public-weba...@w3.org.
--
Anne van Kesteren
http://annevankesteren.nl/
