Re: [whatwg] File API Streaming Blobs
Comments inline...
On Wed, Aug 10, 2011 at 2:05 PM, Charles Pritchard ch...@jumis.com wrote:
On 8/9/2011 9:38 AM, Aaron Colwell wrote:
FYI I'm working on an experimental extension to Chromium to allow media
data to be streamed into a media element via JavaScript. Here is the draft
spechttp://html5-mediasource-api.googlecode.com/svn/tags/0.2/draft-spec/mediasource-draft-spec.html
and
pending WebKit patch https://bugs.webkit.org/show_bug.cgi?id=64731 related
to this work. I have simple WebM VOD playback w/ seeking working where all
media data is fetched via XHR.
It's nice to see this patch.
Thanks. Hopefully I can get it landed soon so people can start playing with
it in Chrome Dev Channel builds.
I'm hoping to see streamed array buffers in XHR, though fetching in chunks
can work,
given the relatively small overhead of HTTP headers vs Video content.
Eventually I'd like to see streamed array buffers in XHR. For now I'm just
using range requests and allowing the JavaScript code determine how large
the ranges should be to control overhead.
The WHAWG specs have a Media Stream example which uses URL createObjectURL:
navigator.getUserMedia('video user', gotStream, noStream);
function gotStream(stream) {
video.src = URL.createObjectURL(stream);
http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html#dom-mediastream
The WHATWG spec seems closer to (mediaElement.createStream()).append()
semantics.
There was a previous discussion about this on WHATWG. There was concern
about providing compressed data to a MediaStream object since they are
basically format agnostic right now.
Both WHATWG and the draft spec agree on src=uri;
The benefit of src=uri is that it allows you to leverage all the existing
state transition and behavior defined in the spec.
File API has to toURL semantics on objects, simlar to the draft spec, for
getting filesystem:// uris.
My understanding: The draft spec is simpler, intended only to be used by
HTMLMediaElement
and only by one element at a time, without introducing a new object. In the
long
run, it may make sense to create a media stream object, consistent with the
WHATWG direction.
The draft spec was intended to be as simple as possible. Attaching this
functionality to HTMLMediaElement instead of
creating a MediaStream came out of discussions on whatwg
herehttp://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2011-July/032283.html
and
herehttp://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2011-July/032384.html.
I'm definitely open to revisiting this, but I got
the feeling that people wanted to see a more concrete implementation first.
I also like having this functionality part of
HTMLMediaElement because then I only have to deal with the HTMLMediaElement
during seeking instead of having to coordinate behavior
between the MediaStream the HTMLMediaElement.
On another note, Mozilla Labs has some experiments on recording video from
canvas (as well as general webcam, etc):
https://mozillalabs.com/rainbow/
https://github.com/mozilla/rainbow
https://github.com/mozilla/rainbow/blob/master/content/example_canvas.html
I'll take a look at this.
Aaron
Re: [whatwg] File API Streaming Blobs
Þann mán 8.ágú 2011 20:31, skrifaði Simon Heckmann: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Has the decryption to happen above the File API?
Re: [whatwg] File API Streaming Blobs
On 8/9/2011 9:38 AM, Aaron Colwell wrote:
FYI I'm working on an experimental extension to Chromium to allow
media data to be streamed into a media element via JavaScript. Here is
the draft spec
http://html5-mediasource-api.googlecode.com/svn/tags/0.2/draft-spec/mediasource-draft-spec.html and
pending WebKit patch
https://bugs.webkit.org/show_bug.cgi?id=64731 related to this work.
I have simple WebM VOD playback w/ seeking working where all media
data is fetched via XHR.
It's nice to see this patch.
I'm hoping to see streamed array buffers in XHR, though fetching in
chunks can work,
given the relatively small overhead of HTTP headers vs Video content.
The WHAWG specs have a Media Stream example which uses URL createObjectURL:
navigator.getUserMedia('video user', gotStream, noStream);
function gotStream(stream) {
video.src = URL.createObjectURL(stream);
http://www.whatwg.org/specs/web-apps/current-work/complete/video-conferencing-and-peer-to-peer-communication.html#dom-mediastream
The WHATWG spec seems closer to (mediaElement.createStream()).append()
semantics.
Both WHATWG and the draft spec agree on src=uri;
File API has to toURL semantics on objects, simlar to the draft spec,
for getting filesystem:// uris.
My understanding: The draft spec is simpler, intended only to be used by
HTMLMediaElement
and only by one element at a time, without introducing a new object. In
the long
run, it may make sense to create a media stream object, consistent with
the WHATWG direction.
On another note, Mozilla Labs has some experiments on recording video
from canvas (as well as general webcam, etc):
https://mozillalabs.com/rainbow/
https://github.com/mozilla/rainbow
https://github.com/mozilla/rainbow/blob/master/content/example_canvas.html
-Charles
Re: [whatwg] File API Streaming Blobs
FYI I'm working on an experimental extension to Chromium to allow media data to be streamed into a media element via JavaScript. Here is the draft spechttp://html5-mediasource-api.googlecode.com/svn/tags/0.2/draft-spec/mediasource-draft-spec.html and pending WebKit patch https://bugs.webkit.org/show_bug.cgi?id=64731 related to this work. I have simple WebM VOD playback w/ seeking working where all media data is fetched via XHR. Aaron On Mon, Aug 8, 2011 at 7:16 PM, Charles Pritchard ch...@jumis.com wrote: On 8/8/2011 2:51 PM, Glenn Maynard wrote: On Mon, Aug 8, 2011 at 4:31 PM, Simon Heckmann si...@simonheckmann.demailto: si...@simonheckmann.de** wrote: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Assuming you're thinking of DRM, are there any related use cases other than crypto? Encryption for DRM, at least, isn't a very compelling use case; client-side Javascript encryption is a very weak level of protection (putting aside, for now, the question of whether the web can or should be attempting to handle DRM in the first place). If it's not DRM you're thinking of, can you clarify? Jonas Sickling brought up a few cases for XHR-based streaming of arraybuffers: progressive rendering of word docs and PDFs. WebP and WebM have had interesting packaging hacks. Packaging itself, whether DRM or not, is compelling. PDF supports embedded data, a wide range of formats. GPAC provides many related tools (MP4 based, I believe): http://gpac.wp.institut-**telecom.fr/http://gpac.wp.institut-telecom.fr/ The audio and video tags drop frames It seems to me that if a listener is not registered to the stream, data would just be dropped. As an alternative, the author could register a fixed length circular buffer. For instance, I could create 1 megabyte arrayview, run URL.createBlobStream(**ArrayView) and use .append(data). That kind of structure may support multicast (multiple audio/video elements) and improved XHR2 semantics. The circular buffer, itself, is easy to prototype: subarray works well with typed arrays. Otherwise relevant, is the work on raw audio data that Firefox and Chromium have released as experimental extensions. It does work on a buffer-based system. -Charles
Re: [whatwg] File API Streaming Blobs
Sorry about the very slow response; I've been on leave, and am now catching up on my email. On Wed, Jun 22, 2011 at 11:54 AM, Arun Ranganathan a...@mozilla.com wrote: Greetings Adam, Ian, I wish I knew that earlier when I originally posted the idea, there was lots of discussion and good ideas but then it suddenly dropped of the face of the earth. Essentially I am fowarding this suggestion to public-weba...@w3.org on the basis as apparently most discussion of File API specs happen there, and would like to know how to move forward with this suggestion. The original suggestion and following comments are on the whatwg list archive, starting with http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029973.html Summing up, the problem with the current implementation of Blobs is that once a URI has been generated for them, by design changes are no longer reflected in the object URL. In a streaming scenario, this is not what is needed, rather a long-living Blob that can be appended is needed and 'streamed' to other parts of the browser, e.g. thevideo oraudio element. The original use case was: make an application which will download media files from a server and cache them locally, as well as playing them without making the user wait for the entire file to be downloaded, converted to a blob, then saved and played, however such an API covers many other use cases such as on-the-fly on-device decryption of streamed media content (ie live streams either without end or static large files that to download completely would be a waste when only the first couple of seconds need to be buffered and decrypted before playback can begin) Some suggestions were to modify or create a new type of Blob, the StreamingBlob which can be changed without its object url changing and appended to as new data is downloaded or decoded, and using a similar process to how large files may start to be decoded/played by a browser before they are fully downloaded. Other suggestions suggested using a pull API on the Blob so browsers can request for new data asynchronously, such as in http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029998.html Some problems however that a browser may face is what to do with urls which are opened twice, and whether the object url should start from the beginning (which would be needed for decoding encrypted, on-demand audio) or start from the end (similar to `tail`, for live streaming events that need decryption, etc.). Thanks, P.S. Sorry if I've not done this the right way by forwarding like this, I'm not usually active on mailing lists. I actually think moving to a streaming mode for file reads in general is desirable, but I'm not entirely sure extending Blobs is the way to go for *that* use case, which honestly is the main use case I'm interested in. We may improve upon ideas after this API goes to Last Call for streaming file reads; hopefully we'll do a better job than other non-JavaScript APIs out there :) [1]. Blob objects as they are currently specified live in memory and represent in memory File objects as well. A change to the underlying file isn't captured in the Blob snapshot; moreover, if the file moves or is no longer present at time of read, an error event is fired while processing a read operation. The object URL may be dereferenced, but will result in a 404. The Streaming API explored by WHATWG uses the Object URL scheme for videoconferencing use cases [2], and so the scheme itself is suitable for resources that are more dynamic than memory-resident Blob objects. Segment-plays/segment dereferencing in general can be handled through media fragments; the scheme can naturally be accompanied by fragment identifiers. I agree that it may be desirable to extend Blobs to do a few other things in general, maybe independent of better file reads. You've Cc'd the right listserv :) I'd be interested in what Eric has to say, since BlobBuilder evolves under his watch. Having reviewed the threads, I'm not absolutely sure that we want to add this stuff to Blob. It seems like streaming is quite a bit different than a lot of the problems people want to solve with Blobs, and we may end up with a bit of a mess if we mash them together. BlobBuilder does seem a decent match as a StreamBuilder, though. Since Blobs are specifically non-mutable, it sounds like what you're looking for is more like createObjectURL(blobBuilder) than createObjectURL(blobBuildler.getBlob()). From the threads and from my head, here are some questions: 1) Would reading from a stream always start at the beginning, or would it start at the current point [e.g. in a live video stream]? 2) Would this have to support infinite streams? 3) Would we be expected to keep around data from the very beginning of a stream, even if e.g. it's a live broadcast and you're now watching hour 7? If not, who controls the buffer size and what's the API for
Re: [whatwg] File API Streaming Blobs
Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Kind regards, Simon Heckmann Am 08.08.2011 um 22:17 schrieb Eric U er...@google.com: Sorry about the very slow response; I've been on leave, and am now catching up on my email. On Wed, Jun 22, 2011 at 11:54 AM, Arun Ranganathan a...@mozilla.com wrote: Greetings Adam, Ian, I wish I knew that earlier when I originally posted the idea, there was lots of discussion and good ideas but then it suddenly dropped of the face of the earth. Essentially I am fowarding this suggestion to public-weba...@w3.org on the basis as apparently most discussion of File API specs happen there, and would like to know how to move forward with this suggestion. The original suggestion and following comments are on the whatwg list archive, starting with http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029973.html Summing up, the problem with the current implementation of Blobs is that once a URI has been generated for them, by design changes are no longer reflected in the object URL. In a streaming scenario, this is not what is needed, rather a long-living Blob that can be appended is needed and 'streamed' to other parts of the browser, e.g. thevideo oraudio element. The original use case was: make an application which will download media files from a server and cache them locally, as well as playing them without making the user wait for the entire file to be downloaded, converted to a blob, then saved and played, however such an API covers many other use cases such as on-the-fly on-device decryption of streamed media content (ie live streams either without end or static large files that to download completely would be a waste when only the first couple of seconds need to be buffered and decrypted before playback can begin) Some suggestions were to modify or create a new type of Blob, the StreamingBlob which can be changed without its object url changing and appended to as new data is downloaded or decoded, and using a similar process to how large files may start to be decoded/played by a browser before they are fully downloaded. Other suggestions suggested using a pull API on the Blob so browsers can request for new data asynchronously, such as in http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029998.html Some problems however that a browser may face is what to do with urls which are opened twice, and whether the object url should start from the beginning (which would be needed for decoding encrypted, on-demand audio) or start from the end (similar to `tail`, for live streaming events that need decryption, etc.). Thanks, P.S. Sorry if I've not done this the right way by forwarding like this, I'm not usually active on mailing lists. I actually think moving to a streaming mode for file reads in general is desirable, but I'm not entirely sure extending Blobs is the way to go for *that* use case, which honestly is the main use case I'm interested in. We may improve upon ideas after this API goes to Last Call for streaming file reads; hopefully we'll do a better job than other non-JavaScript APIs out there :) [1]. Blob objects as they are currently specified live in memory and represent in memory File objects as well. A change to the underlying file isn't captured in the Blob snapshot; moreover, if the file moves or is no longer present at time of read, an error event is fired while processing a read operation. The object URL may be dereferenced, but will result in a 404. The Streaming API explored by WHATWG uses the Object URL scheme for videoconferencing use cases [2], and so the scheme itself is suitable for resources that are more dynamic than memory-resident Blob objects. Segment-plays/segment dereferencing in general can be handled through media fragments; the scheme can naturally be accompanied by fragment identifiers. I agree that it may be desirable to extend Blobs to do a few other things in general, maybe independent of better file reads. You've Cc'd the right listserv :) I'd be interested in what Eric has to say, since BlobBuilder evolves under his watch. Having reviewed the threads, I'm not absolutely sure that we want to add this stuff to Blob. It seems like streaming is quite a bit different than a lot of the problems people want to solve with Blobs, and we may end up with a bit of a mess if we mash them together. BlobBuilder does seem a decent match as a StreamBuilder, though. Since Blobs are specifically non-mutable, it sounds like what you're looking for is more like
Re: [whatwg] File API Streaming Blobs
On Mon, Aug 8, 2011 at 4:31 PM, Simon Heckmann si...@simonheckmann.dewrote: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Assuming you're thinking of DRM, are there any related use cases other than crypto? Encryption for DRM, at least, isn't a very compelling use case; client-side Javascript encryption is a very weak level of protection (putting aside, for now, the question of whether the web can or should be attempting to handle DRM in the first place). If it's not DRM you're thinking of, can you clarify? -- Glenn Maynard
Re: [whatwg] File API Streaming Blobs
It's actually confidential company data, I was thinking off. Together with the DOMCrypt API I thought this could be a valid use case. But I think there might be more cases in which it might make sense to preprocess locally stored video data. Kind regards, Simon Heckmann Am 08.08.2011 um 23:51 schrieb Glenn Maynard gl...@zewt.org: On Mon, Aug 8, 2011 at 4:31 PM, Simon Heckmann si...@simonheckmann.de wrote: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Assuming you're thinking of DRM, are there any related use cases other than crypto? Encryption for DRM, at least, isn't a very compelling use case; client-side Javascript encryption is a very weak level of protection (putting aside, for now, the question of whether the web can or should be attempting to handle DRM in the first place). If it's not DRM you're thinking of, can you clarify? -- Glenn Maynard
Re: [whatwg] File API Streaming Blobs
Sorry to jump in the middle of your discussion but after reading Eric's questions e.g. I haven't fully absorbed the MediaStream API, but perhaps it would be more natural to make a connector in that API rather than modifying Blob? I think this use case also applies for other stream/file/blob analysis and processing e.g. Augmented Reality, Object Recognition, DSP, etc. I've raised this recently across the related groups [1] with a simple use case and a number of supporting requirements. roBman [1] http://lists.w3.org/Archives/Public/public-webrtc/2011Jul/0170.html On Mon, 2011-08-08 at 23:59 +0200, Simon Heckmann wrote: It's actually confidential company data, I was thinking off. Together with the DOMCrypt API I thought this could be a valid use case. But I think there might be more cases in which it might make sense to preprocess locally stored video data. Kind regards, Simon Heckmann Am 08.08.2011 um 23:51 schrieb Glenn Maynard gl...@zewt.org: On Mon, Aug 8, 2011 at 4:31 PM, Simon Heckmann si...@simonheckmann.de wrote: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Assuming you're thinking of DRM, are there any related use cases other than crypto? Encryption for DRM, at least, isn't a very compelling use case; client-side Javascript encryption is a very weak level of protection (putting aside, for now, the question of whether the web can or should be attempting to handle DRM in the first place). If it's not DRM you're thinking of, can you clarify? -- Glenn Maynard
Re: [whatwg] File API Streaming Blobs
On 8/8/2011 2:51 PM, Glenn Maynard wrote: On Mon, Aug 8, 2011 at 4:31 PM, Simon Heckmann si...@simonheckmann.de mailto:si...@simonheckmann.de wrote: Well, not directly an answer to your question, but the use case I had in mind is the following: A large encrypted video (e.g. HD movie with 2GB) file is stored using the File API, I then want to decrypt this file and start playing with only a minor delay. I do not want to decrypt the entire file before it can be viewed. As long as such as use case gets covered I am fine with everything. Assuming you're thinking of DRM, are there any related use cases other than crypto? Encryption for DRM, at least, isn't a very compelling use case; client-side Javascript encryption is a very weak level of protection (putting aside, for now, the question of whether the web can or should be attempting to handle DRM in the first place). If it's not DRM you're thinking of, can you clarify? Jonas Sickling brought up a few cases for XHR-based streaming of arraybuffers: progressive rendering of word docs and PDFs. WebP and WebM have had interesting packaging hacks. Packaging itself, whether DRM or not, is compelling. PDF supports embedded data, a wide range of formats. GPAC provides many related tools (MP4 based, I believe): http://gpac.wp.institut-telecom.fr/ The audio and video tags drop frames It seems to me that if a listener is not registered to the stream, data would just be dropped. As an alternative, the author could register a fixed length circular buffer. For instance, I could create 1 megabyte arrayview, run URL.createBlobStream(ArrayView) and use .append(data). That kind of structure may support multicast (multiple audio/video elements) and improved XHR2 semantics. The circular buffer, itself, is easy to prototype: subarray works well with typed arrays. Otherwise relevant, is the work on raw audio data that Firefox and Chromium have released as experimental extensions. It does work on a buffer-based system. -Charles
Re: [whatwg] File API Streaming Blobs
Greetings Adam, Ian, I wish I knew that earlier when I originally posted the idea, there was lots of discussion and good ideas but then it suddenly dropped of the face of the earth. Essentially I am fowarding this suggestion to public-weba...@w3.org on the basis as apparently most discussion of File API specs happen there, and would like to know how to move forward with this suggestion. The original suggestion and following comments are on the whatwg list archive, starting with http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029973.html Summing up, the problem with the current implementation of Blobs is that once a URI has been generated for them, by design changes are no longer reflected in the object URL. In a streaming scenario, this is not what is needed, rather a long-living Blob that can be appended is needed and 'streamed' to other parts of the browser, e.g. thevideo oraudio element. The original use case was: make an application which will download media files from a server and cache them locally, as well as playing them without making the user wait for the entire file to be downloaded, converted to a blob, then saved and played, however such an API covers many other use cases such as on-the-fly on-device decryption of streamed media content (ie live streams either without end or static large files that to download completely would be a waste when only the first couple of seconds need to be buffered and decrypted before playback can begin) Some suggestions were to modify or create a new type of Blob, the StreamingBlob which can be changed without its object url changing and appended to as new data is downloaded or decoded, and using a similar process to how large files may start to be decoded/played by a browser before they are fully downloaded. Other suggestions suggested using a pull API on the Blob so browsers can request for new data asynchronously, such as in http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029998.html Some problems however that a browser may face is what to do with urls which are opened twice, and whether the object url should start from the beginning (which would be needed for decoding encrypted, on-demand audio) or start from the end (similar to `tail`, for live streaming events that need decryption, etc.). Thanks, P.S. Sorry if I've not done this the right way by forwarding like this, I'm not usually active on mailing lists. I actually think moving to a streaming mode for file reads in general is desirable, but I'm not entirely sure extending Blobs is the way to go for *that* use case, which honestly is the main use case I'm interested in. We may improve upon ideas after this API goes to Last Call for streaming file reads; hopefully we'll do a better job than other non-JavaScript APIs out there :) [1]. Blob objects as they are currently specified live in memory and represent in memory File objects as well. A change to the underlying file isn't captured in the Blob snapshot; moreover, if the file moves or is no longer present at time of read, an error event is fired while processing a read operation. The object URL may be dereferenced, but will result in a 404. The Streaming API explored by WHATWG uses the Object URL scheme for videoconferencing use cases [2], and so the scheme itself is suitable for resources that are more dynamic than memory-resident Blob objects. Segment-plays/segment dereferencing in general can be handled through media fragments; the scheme can naturally be accompanied by fragment identifiers. I agree that it may be desirable to extend Blobs to do a few other things in general, maybe independent of better file reads. You've Cc'd the right listserv :) I'd be interested in what Eric has to say, since BlobBuilder evolves under his watch. -- A* [1] http://download.oracle.com/javase/1.4.2/docs/api/java/io/FileInputStream.html [2] http://www.whatwg.org/specs/web-apps/current-work/#stream-api
Re: [whatwg] File API Streaming Blobs
Ian, I wish I knew that earlier when I originally posted the idea, there was lots of discussion and good ideas but then it suddenly dropped of the face of the earth. Essentially I am fowarding this suggestion to public-weba...@w3.org on the basis as apparently most discussion of File API specs happen there, and would like to know how to move forward with this suggestion. The original suggestion and following comments are on the whatwg list archive, starting with http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029973.html Summing up, the problem with the current implementation of Blobs is that once a URI has been generated for them, by design changes are no longer reflected in the object URL. In a streaming scenario, this is not what is needed, rather a long-living Blob that can be appended is needed and 'streamed' to other parts of the browser, e.g. the video or audio element. The original use case was: make an application which will download media files from a server and cache them locally, as well as playing them without making the user wait for the entire file to be downloaded, converted to a blob, then saved and played, however such an API covers many other use cases such as on-the-fly on-device decryption of streamed media content (ie live streams either without end or static large files that to download completely would be a waste when only the first couple of seconds need to be buffered and decrypted before playback can begin) Some suggestions were to modify or create a new type of Blob, the StreamingBlob which can be changed without its object url changing and appended to as new data is downloaded or decoded, and using a similar process to how large files may start to be decoded/played by a browser before they are fully downloaded. Other suggestions suggested using a pull API on the Blob so browsers can request for new data asynchronously, such as in http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/029998.html Some problems however that a browser may face is what to do with urls which are opened twice, and whether the object url should start from the beginning (which would be needed for decoding encrypted, on-demand audio) or start from the end (similar to `tail`, for live streaming events that need decryption, etc.). Thanks, P.S. Sorry if I've not done this the right way by forwarding like this, I'm not usually active on mailing lists. On Wed, Jun 15, 2011 at 9:59 AM, Ian Hickson i...@hixie.ch wrote: On Mon, 21 Mar 2011, Simon Heckmann wrote: I found this thread (http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/02997 3.html) in the archive of this mailing list, but I could not determine if any decision was made whether this should be implemented or not. I am interested in this, because I came up with a scenario which would benefit from Streaming Blobs: I use the File API to locally store larger video files on the users hard drive. For security purposes I encrypted them with AES and use a javascript library for local just-in-time decryption. This is not yet a productive application bur more of a prototype. However, I experienced javascript manipulation of large data can be quite slow so we do not want the user to wait until the full video is decrypted/manipulated. Therefore I would vote for a way to append data to a Blob and the ObjectURL reflects these modifications. Maybe something like createStreamableObjectURL could be used for differentiation? Just wanted to express my thoughts because I think the whole File API is a great idea! I recommend forwarding your suggestion to the W3C public-weba...@w3.org mailing list, which is where discussion of the File API specs more usually takes place. -- Ian Hickson U+1047E )\._.,--,'``. fL http://ln.hixie.ch/ U+263A /, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.' -- Adam Malcontenti-Wilson
Re: [whatwg] File API Streaming Blobs
On Mon, 21 Mar 2011, Simon Heckmann wrote: I found this thread (http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/02997 3.html) in the archive of this mailing list, but I could not determine if any decision was made whether this should be implemented or not. I am interested in this, because I came up with a scenario which would benefit from Streaming Blobs: I use the File API to locally store larger video files on the users hard drive. For security purposes I encrypted them with AES and use a javascript library for local just-in-time decryption. This is not yet a productive application bur more of a prototype. However, I experienced javascript manipulation of large data can be quite slow so we do not want the user to wait until the full video is decrypted/manipulated. Therefore I would vote for a way to append data to a Blob and the ObjectURL reflects these modifications. Maybe something like createStreamableObjectURL could be used for differentiation? Just wanted to express my thoughts because I think the whole File API is a great idea! I recommend forwarding your suggestion to the W3C public-weba...@w3.org mailing list, which is where discussion of the File API specs more usually takes place. -- Ian Hickson U+1047E)\._.,--,'``.fL http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,. Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] File API Streaming Blobs
Hello everyone, I found this thread (http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2011-January/02997 3.html) in the archive of this mailing list, but I could not determine if any decision was made whether this should be implemented or not. I am interested in this, because I came up with a scenario which would benefit from Streaming Blobs: I use the File API to locally store larger video files on the users hard drive. For security purposes I encrypted them with AES and use a javascript library for local just-in-time decryption. This is not yet a productive application bur more of a prototype. However, I experienced javascript manipulation of large data can be quite slow so we do not want the user to wait until the full video is decrypted/manipulated. Therefore I would vote for a way to append data to a Blob and the ObjectURL reflects these modifications. Maybe something like createStreamableObjectURL could be used for differentiation? Just wanted to express my thoughts because I think the whole File API is a great idea! Kind regards, Simon Heckmann
Re: [whatwg] File API Streaming Blobs
On Tue, 25 Jan 2011 19:05:32 +0100, David Flanagan da...@davidflanagan.com wrote: I don't see the point of that, if all it does is save one call to URL.createObjectURL() (and also one call to revokeObjectURL())? The point is that authors will not have to do any cleanup. Objects get cleaned up automatically. So besides being easier it is also safer. Adam's use case--to be able to download, play and cache audio data at the same time--seems like a pretty compelling one. I think this is fundamentally an issue with the Blob API, not the URL API. Blobs just seem like they ought to stream. When you get a blob in the onprogress handler of an XHR2, you ought to be able to fire up a FileReader on it and have it automatically read from the blob as the XHR2 writes to the blob. But currently (I think) you have to slice the blob to get only the new bytes and start a new FileReader each time onprogress is called. (Or wait for onload, of course.) You have to wait for onload currently. Similarly, when you get your first onprogress event for a Blob download, you ought to be able to create a URL for that Blob that remains valid while the download is in progress. So you can use that url with an audio element, for example. Yeah, I do not quite see why the URL cannot remain valid. P.S. This is probably the wrong list for this discussion, isn't it? public-weba...@w3.org is preferable, but it does not matter much. -- Anne van Kesteren http://annevankesteren.nl/
Re: [whatwg] File API Streaming Blobs
On 01/24/2011 04:24 AM, Anne van Kesteren wrote: (I removed the Chromium related list as I am not subscribed there.) On Fri, 21 Jan 2011 21:35:53 +0100, Adam Malcontenti-Wilson adman@gmail.com wrote: XHR2 is one part of the APIs required for my use case as that is the easiest way to download for example a music file. However, while downloading, there's no way to pipe the download(ing) blob to the HTML5 Audio element as to play Audio it requires an object URL, and an object URL can (currently) only point to a static Blob, as well as the fact that a Blob cannot be appended. This would be important for listening streaming audio that needs to be processed in JavaScript or cached to persistant storage using the Filesystem APIs without having to wait for the entire file to be downloaded into an ArrayBuffer or Blob. There is a plan of allowing direct assigning to IDL attributes besides creating URLs. I.e. being able to do: audio.src = blob (The src content attribute would then be something like about:objecturl.) I am not sure if that API should work differently from creating URLs and assigning those, but we could consider it. I don't see the point of that, if all it does is save one call to URL.createObjectURL() (and also one call to revokeObjectURL())? In any case, making this behave differently than the URL API seems like a bad idea. My suggestion was for another alternative version of Blob and/or createObjectUrl that mimicks how a HTTP request can be parsed and (in the case of audio or video) start playing before it has finished downloading (e.g. got to the content-length or had a connection close) by pushing content when it is appended to the blob and then the virtual connection can be closed when the Blob has finished being built by calling a close() function. I've also thought of other alternatives, but I'd make sure that there isn't already a way to do this with the current (specified) APIs, and I think this has the most other use cases as any data that takes a while to process can be streamed to the user or other parts of the browser. Is there actually a good reason for the URL API to have behave in this way? Adam's use case--to be able to download, play and cache audio data at the same time--seems like a pretty compelling one. I think this is fundamentally an issue with the Blob API, not the URL API. Blobs just seem like they ought to stream. When you get a blob in the onprogress handler of an XHR2, you ought to be able to fire up a FileReader on it and have it automatically read from the blob as the XHR2 writes to the blob. But currently (I think) you have to slice the blob to get only the new bytes and start a new FileReader each time onprogress is called. (Or wait for onload, of course.) Similarly, when you get your first onprogress event for a Blob download, you ought to be able to create a URL for that Blob that remains valid while the download is in progress. So you can use that url with an audio element, for example. Also: BlobBuilder seems to me as if it ought to be a streaming API. It feels like it ought to work like this: var bb = new BlobBuilder(); var b1 = bb.getBlob(); var u1 = URL.createObjectURL(b1); bb.append(hello world); var b2 = bb.getBlob(); var u2 = URL.createObjectURL(b2); b1 === b2 // They ought to be equal, but they're not u1 === u2 // Ought to be equal, but they're not bb.close(); // New method: now no more appends are allowed. David P.S. This is probably the wrong list for this discussion, isn't it?
Re: [whatwg] File API Streaming Blobs
On 1/21/11, Roger Hågensen resca...@emsai.net wrote: Hmm! And I guess it's very difficult to create a abstract in/out interface that can handle any protocol/stream. Although an abstract in/out would be ideal as that would let new protocols to be supported without needing to rewrite anything at the higher level. stdio2?
Re: [whatwg] File API Streaming Blobs
(I removed the Chromium related list as I am not subscribed there.) On Fri, 21 Jan 2011 21:35:53 +0100, Adam Malcontenti-Wilson adman@gmail.com wrote: XHR2 is one part of the APIs required for my use case as that is the easiest way to download for example a music file. However, while downloading, there's no way to pipe the download(ing) blob to the HTML5 Audio element as to play Audio it requires an object URL, and an object URL can (currently) only point to a static Blob, as well as the fact that a Blob cannot be appended. This would be important for listening streaming audio that needs to be processed in JavaScript or cached to persistant storage using the Filesystem APIs without having to wait for the entire file to be downloaded into an ArrayBuffer or Blob. There is a plan of allowing direct assigning to IDL attributes besides creating URLs. I.e. being able to do: audio.src = blob (The src content attribute would then be something like about:objecturl.) I am not sure if that API should work differently from creating URLs and assigning those, but we could consider it. My suggestion was for another alternative version of Blob and/or createObjectUrl that mimicks how a HTTP request can be parsed and (in the case of audio or video) start playing before it has finished downloading (e.g. got to the content-length or had a connection close) by pushing content when it is appended to the blob and then the virtual connection can be closed when the Blob has finished being built by calling a close() function. I've also thought of other alternatives, but I'd make sure that there isn't already a way to do this with the current (specified) APIs, and I think this has the most other use cases as any data that takes a while to process can be streamed to the user or other parts of the browser. Is there actually a good reason for the URL API to have behave in this way? -- Anne van Kesteren http://annevankesteren.nl/
Re: [whatwg] File API Streaming Blobs
Doesn't the current XHR2 spec address this use case?
Browsers don't seem to implement it yet, but shouldn't something like
this work for the original poster?
x = new XMLHttpRequest()
x.open(GET, http://my-media-file;);
x.responseType = blob;
x.send();
var nbytes = 0;
x.onprogress = function(e) {
var blob = x.response.slice(nbytes, e.loaded-nbytes);
nbytes += blob.size;
var reader = new FileReader();
reader.readAsArrayBuffer(blob, function() {
// process blob content here
});
}
David Flanagan
On 01/21/2011 02:02 AM, Jeremy Orlow wrote:
Would something like this tie in to thedevice work that's being done
maybe?
-- Forwarded message --
From: Adam Malcontenti-Wilsonadman@gmail.com
Date: Fri, Jan 21, 2011 at 6:21 AM
Subject: [chromium-html5] File API Streaming Blobs
To: Chromium HTML5chromium-ht...@chromium.org
Hi.
I'm trying to make an application which will download media files from
a server and cache them locally, as well as playing them back but I'm
trying to figure out how I could do so without making the user wait
for the entire file to be downloaded, converted to a blob, then
saved.
For example, suppose that I create a new BlobBuilder, append hello,
get the Blob, and then create an object url from that blob, and then
open the object url. Any other text that I append to the BlobBuilder
would not go into the old blob that I created a url for, and hence not
shown making streaming impossible.
Is there any other methods in the spec(s) to implement such
streaming?
If not, perhaps there needs to be yet another object to have a way of
creating a StreamingBlob that doesn't close the virtual connection
to the browser until a close method is called, thereby facilitating
streaming.
Thanks,
Re: [whatwg] File API Streaming Blobs
XHR2 is one part of the APIs required for my use case as that is the
easiest way to download for example a music file. However, while
downloading, there's no way to pipe the download(ing) blob to the
HTML5 Audio element as to play Audio it requires an object URL, and an
object URL can (currently) only point to a static Blob, as well as the
fact that a Blob cannot be appended. This would be important for
listening streaming audio that needs to be processed in JavaScript or
cached to persistant storage using the Filesystem APIs without having
to wait for the entire file to be downloaded into an ArrayBuffer or
Blob.
I'm not really sure about the device tag fitting in with this as it
seems to be more about reading serial ports and cameras from what I've
seen, however any streaming interface that could be specified could be
also used in the device API to stream binary content to devices.
My suggestion was for another alternative version of Blob and/or
createObjectUrl that mimicks how a HTTP request can be parsed and (in
the case of audio or video) start playing before it has finished
downloading (e.g. got to the content-length or had a connection close)
by pushing content when it is appended to the blob and then the
virtual connection can be closed when the Blob has finished being
built by calling a close() function. I've also thought of other
alternatives, but I'd make sure that there isn't already a way to do
this with the current (specified) APIs, and I think this has the most
other use cases as any data that takes a while to process can be
streamed to the user or other parts of the browser.
Thanks,
--
Adam Malcontenti-Wilson
On Sat, Jan 22, 2011 at 5:55 AM, David Flanagan da...@davidflanagan.com wrote:
Doesn't the current XHR2 spec address this use case?
Browsers don't seem to implement it yet, but shouldn't something like this
work for the original poster?
x = new XMLHttpRequest()
x.open(GET, http://my-media-file;);
x.responseType = blob;
x.send();
var nbytes = 0;
x.onprogress = function(e) {
var blob = x.response.slice(nbytes, e.loaded-nbytes);
nbytes += blob.size;
var reader = new FileReader();
reader.readAsArrayBuffer(blob, function() {
// process blob content here
});
}
David Flanagan
On 01/21/2011 02:02 AM, Jeremy Orlow wrote:
Would something like this tie in to thedevice work that's being done
maybe?
-- Forwarded message --
From: Adam Malcontenti-Wilsonadman@gmail.com
Date: Fri, Jan 21, 2011 at 6:21 AM
Subject: [chromium-html5] File API Streaming Blobs
To: Chromium HTML5chromium-ht...@chromium.org
Hi.
I'm trying to make an application which will download media files from
a server and cache them locally, as well as playing them back but I'm
trying to figure out how I could do so without making the user wait
for the entire file to be downloaded, converted to a blob, then
saved.
For example, suppose that I create a new BlobBuilder, append hello,
get the Blob, and then create an object url from that blob, and then
open the object url. Any other text that I append to the BlobBuilder
would not go into the old blob that I created a url for, and hence not
shown making streaming impossible.
Is there any other methods in the spec(s) to implement such
streaming?
If not, perhaps there needs to be yet another object to have a way of
creating a StreamingBlob that doesn't close the virtual connection
to the browser until a close method is called, thereby facilitating
streaming.
Thanks,
--
Adam Malcontenti-Wilson
Re: [whatwg] File API Streaming Blobs
On Fri, Jan 21, 2011 at 1:55 PM, David Flanagan da...@davidflanagan.com wrote:
Doesn't the current XHR2 spec address this use case?
Browsers don't seem to implement it yet, but shouldn't something like this
work for the original poster?
x = new XMLHttpRequest()
x.open(GET, http://my-media-file;);
x.responseType = blob;
x.send();
var nbytes = 0;
x.onprogress = function(e) {
var blob = x.response.slice(nbytes, e.loaded-nbytes);
nbytes += blob.size;
var reader = new FileReader();
reader.readAsArrayBuffer(blob, function() {
// process blob content here
});
}
He wants to be able to stream data out, not just in.
It's tricky in practice, because there's no way for whoever's reading
the stream to block. For example, if you're reading a 1 GB video on a
phone with 256 MB of memory, it needs to stop buffering when it's out
of memory until some data has been played and thrown away, as it would
when streaming from the network normally. That requiests an API more
complex than simply writing to a file.
--
Glenn Maynard
Re: [whatwg] File API Streaming Blobs
On 2011-01-21 21:50, Glenn Maynard wrote: On Fri, Jan 21, 2011 at 1:55 PM, David Flanaganda...@davidflanagan.com wrote: Doesn't the current XHR2 spec address this use case? Browsers don't seem to implement it yet, but shouldn't something like this work for the original poster? He wants to be able to stream data out, not just in. It's tricky in practice, because there's no way for whoever's reading the stream to block. For example, if you're reading a 1 GB video on a phone with 256 MB of memory, it needs to stop buffering when it's out of memory until some data has been played and thrown away, as it would when streaming from the network normally. That requiests an API more complex than simply writing to a file. Hmm! And I guess it's very difficult to create a abstract in/out interface that can handle any protocol/stream. Although an abstract in/out would be ideal as that would let new protocols to be supported without needing to rewrite anything at the higher level. -- Roger Rescator Hågensen. Freelancer - http://www.EmSai.net/
Re: [whatwg] File API Streaming Blobs
On Fri, Jan 21, 2011 at 4:59 PM, Roger Hågensen resca...@emsai.net wrote:
Hmm! And I guess it's very difficult to create a abstract in/out interface
that can handle any protocol/stream.
Although an abstract in/out would be ideal as that would let new protocols
to be supported without needing to rewrite anything at the higher level.
I wouldn't expect it to try to pretend it's a network protocol any
more than file:// URLs.
Another complexity: what happens if you create a URL for a streaming
blob, and then it's opened more than once? That might happen even if
you only use the URL in one place.
It would need to be a pull API, not a push API, with a callback to
request blocks of data asynchronously. Here's an example of how that
could work. Notice that the response to b.requestData is async: the
response callbacks can be called after requestData returns. This
implicitly handles cases where the URL is opened multiple times,
allows seeking and caller bandwidth throttling. (This is just a rough
sketch, of course; there'd need to be mechanisms for EOF, for
providing the total file size if known so download progress bars work,
and so on.)
var b = new StreamingBlob();
b.requestData = function(request, response)
{
var blob = dataBlob.slice(request.start, request.bytes);
var reader = new FileReader();
reader.onload = function(e) { response.success(reader.result); }
reader.onerror = function(e) { response.error(...); }
reader.readAsBinaryString(blob);
}
video.src = createObjectURL(b);
--
Glenn Maynard
