Re: [whatwg] Access to live/raw audio and video stream data from both local and remote sources
Rob, I'm sorry for the late answer. The W3C DAP and WebRTC chairs have discussed this, and come to the following: - The WebRTC WG deals with access to live (audio and video) streams, and also currently have support for local recording of them in the API proposal [1]. - DAP has a note about the device element in the HTML Media Capture draft, but the device element has been replaced by getUserMedia [1]. - In the WebRTC charter there are references to DAP regarding device exploration and media capturing as that was deemed as in DAP scope at the time of writing the WebRTC charter. This has however since been resolved, for media streams this will be handled by WebRTC. - WebRTC is planning coordination with the Audio WG to ensure alignment regarding media streams. A question: what do you mean by raw audio and video stream data? The MediaStreams discussed in WebRTC are more of logical references (which you can attach to audio/video elements for rendering, to a PeerConnection for streaming to a peer and so on). Stefan (for the DAP and WebRTC chairs). [1] http://dev.w3.org/2011/webrtc/editor/webrtc.html On 2011-07-27 02:56, Rob Manson wrote: Hi, sorry for posting across multiple groups, but I hope you'll see from my comments below that this is really needed. This is definitely not intended as criticism of any of the work going on. It's intended as constructive feedback that hopefully provides clarification on a key use case and it's supporting requirements. Access to live/raw audio and video stream data from both local and remote sources in a consistent way I've spent quite a bit of time trying to follow a clear thread of requirements/solutions that provide API access to raw stream data (e.g. audio, video, etc.). But I'm a bit concerned this is falling in the gap between the DAP and RTC WGs. If this is not the case then please point me to the relevant docs and I'll happily get back in my box 8) Here's how the thread seems to flow at the moment based on public documents. On the DAP page [1] the mission states: the Device APIs and Policy Working Group is to create client-side APIs that enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc So it seems clear that this is the place to start. Further down that page the HTML Media Capture and Media Capture APIs are listed. HTML Media Capture (camera/microphone interactions through HTML forms) initially seems like a good candidate, however the intro in the latest PWD [2] clearly states: Providing streaming access to these capabilities is outside of the scope of this specification. Followed by a NOTE that states: The Working Group is investigating the opportunity to specify streaming access via the proposeddevice element. The link on the proposeddevice element [3] links to a no longer maintained document that then redirects to the top level of the whatwg current work page [4]. On that page the most relevant link is the video conferencing and peer-to-peer communication section [5]. More about that further below. So back to the DAP page to follow explore the other Media Capture API (programmatic access to camera/microphone) [1] and it's latest PWD [6]. The abstract states: This specification defines an Application Programming Interface (API) that provides access to the audio, image and video capture capabilities of the device. And the introduction states: The Capture API defines a high-level interface for accessing the microphone and camera of a hosting device. It completes the HTML Form Based Media Capturing specification [HTMLMEDIACAPTURE] with a programmatic access to start a parametrized capture process. So it seems clear that this is not related to streams in any way either. The Notes column for this API on the DAP page [1] also states: Programmatic API that completes the form based approach Need to check if still interest in this How does it relate with the Web RTC Working Group? Is there an updated position on this? So if you then head over to the WebRTC WG's charter [7] it states: ...to define client-side APIs to enable Real-Time Communications in Web browsers. These APIs should enable building applications that can be run inside a browser, requiring no extra downloads or plugins, that allow communication between parties using audio, video and supplementary real-time communication, without having to use intervening servers... So this is clearly focused upon peer-to-peer communication between systems and the stream related access is naturally just treated as an ancillary requirement. The scope section then states: Enabling real-time communications between Web browsers require
Re: [whatwg] Access to live/raw audio and video stream data from both local and remote sources
Hi Stefan, thanks for your reply and following up...this all sounds good. Especially the alignment with the Audio WG. I'd love to see some resolution on the Web Audio vs Audio Data proposals. By raw I was just clumsily referring to being able to access the data within the tracks within the stream programmatically. Here's a very clumsy example I created that shows the type of access I mean http://robman.com.au/using-live-audio-data-to-drive-your-css-in-ht Obviously this uses the non-standard Mozilla Audio Data API. At the moment you can do a similar-ish thing with video by pumping it into a canvas element then using getImageData...but this is a bit of a clumsy and inefficient pipeline. Ideally we could programmatically reach into the stream data of any of the video and audio tracks to process the data in any number of ways including signal analysis, image recognition, dynamic interface creation, etc. In fact I'm presenting at a conference on Friday on just that topic http://uxaustralia.com.au/conference-2011/is-that-sense-able I think the Khronos StreamInput work has this type of thing in mind too so I wonder if that's also been discussed within the RTC/DAP/Audio WGs? roBman PS: I am really concerned about being labelled as a cross posting spam king so I apologise for creating any of these cross posting storms. However, it really seems like the best way to make sure that this cross-group discussion is shared with all the relevant groups. On Wed, 2011-08-24 at 08:39 +0200, Stefan Håkansson LK wrote: Rob, I'm sorry for the late answer. The W3C DAP and WebRTC chairs have discussed this, and come to the following: - The WebRTC WG deals with access to live (audio and video) streams, and also currently have support for local recording of them in the API proposal [1]. - DAP has a note about the device element in the HTML Media Capture draft, but the device element has been replaced by getUserMedia [1]. - In the WebRTC charter there are references to DAP regarding device exploration and media capturing as that was deemed as in DAP scope at the time of writing the WebRTC charter. This has however since been resolved, for media streams this will be handled by WebRTC. - WebRTC is planning coordination with the Audio WG to ensure alignment regarding media streams. A question: what do you mean by raw audio and video stream data? The MediaStreams discussed in WebRTC are more of logical references (which you can attach to audio/video elements for rendering, to a PeerConnection for streaming to a peer and so on). Stefan (for the DAP and WebRTC chairs). [1] http://dev.w3.org/2011/webrtc/editor/webrtc.html On 2011-07-27 02:56, Rob Manson wrote: Hi, sorry for posting across multiple groups, but I hope you'll see from my comments below that this is really needed. This is definitely not intended as criticism of any of the work going on. It's intended as constructive feedback that hopefully provides clarification on a key use case and it's supporting requirements. Access to live/raw audio and video stream data from both local and remote sources in a consistent way I've spent quite a bit of time trying to follow a clear thread of requirements/solutions that provide API access to raw stream data (e.g. audio, video, etc.). But I'm a bit concerned this is falling in the gap between the DAP and RTC WGs. If this is not the case then please point me to the relevant docs and I'll happily get back in my box 8) Here's how the thread seems to flow at the moment based on public documents. On the DAP page [1] the mission states: the Device APIs and Policy Working Group is to create client-side APIs that enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc So it seems clear that this is the place to start. Further down that page the HTML Media Capture and Media Capture APIs are listed. HTML Media Capture (camera/microphone interactions through HTML forms) initially seems like a good candidate, however the intro in the latest PWD [2] clearly states: Providing streaming access to these capabilities is outside of the scope of this specification. Followed by a NOTE that states: The Working Group is investigating the opportunity to specify streaming access via the proposeddevice element. The link on the proposeddevice element [3] links to a no longer maintained document that then redirects to the top level of the whatwg current work page [4]. On that page the most relevant link is the video conferencing and peer-to-peer communication section [5]. More about that further below. So back to the DAP page to follow explore the other Media Capture API (programmatic access to camera/microphone) [1] and it's latest PWD
[whatwg] add html-attribute for responsive images
Hi all, Paul Irish said I should mention my problems here. So I will: As we now have the possibility of creating fluid and responsive layouts in several ways we have a problem with images. There's currently no good feature to implement something like responsive images which adapt to the different device-resolutions. We only can implement one image with one resolution scaling-up and down. Of course the best solution would be to have a file format supporting this feature by offering diff. quality like .sid format does. But it's very unlikely to have such a feature in the next few years. So I thought of a simple html-attribute to work with media-queries and something like media-size-sources to provide different images fitting to the device-resolution. New browsers can read the new attributes but even old browsers degrade gracefully with the basic src attrobute. I shared the code in a gist here: https://gist.github.com/1158855 and will write it down here, too: !-- backwards compatible solution -- !-- the browser should read media-query and then download the right file for the actual screen size -- img src=http://cdn.url.com/img/myimage_xs.jpg; media-xs=(min-device-width:320px and max-device-width:640px) media-src-xs=http://cdn.url.com/img/myimage_xs.jpg; media-m=(min-device-width:640px and max-device-width:1024px) media-src-m=http://cdn.url.com/img/myimage_m.jpg; media-xl=(min-device-width:1024px) media-src-xl=http://cdn.url.com/img/myimage_xsl.jpg; / Tried several other ways, too, (you can see in versions) but this seems to be the best. What do you think about providing a way to use responsive media through media-queries? For now there are only some JS-Cookie solutions which are not a good approach and are very unflexible. kindest regards, -Anselm
Re: [whatwg] adding microdata to basic links
On Wed, Aug 24, 2011 at 10:02 AM, Stéphane Corlosquet scorlosq...@gmail.com wrote: Starting from a basic markup like this: [[[ This book has been authored by a href=http://smith.org/john;John Smith/a. ]]] I would like to markup both the textContent of the link (John Smith) and the url from the href attribute. In RDFa this is done by adding a couple of attributes to the a element. It would read like this: [[[ This book has been authored by a property=name rel=url href= http://smith.org/john;John Smith/a. ]]] Is there any way to do the same in microdata without adding a new HTML element to the markup? No, Microdata purposely keeps its data model simple by expressing property names through a single attribute. Since having @itemprop on an a always refers to the @href of the element, you must nest an additional element, such as a span, into your markup to carry the property that refers to the text content. ~TJ
Re: [whatwg] adding microdata to basic links
On Wed, Aug 24, 2011 at 1:18 PM, Tab Atkins Jr. jackalm...@gmail.comwrote: On Wed, Aug 24, 2011 at 10:02 AM, Stéphane Corlosquet scorlosq...@gmail.com wrote: Starting from a basic markup like this: [[[ This book has been authored by a href=http://smith.org/john;John Smith/a. ]]] I would like to markup both the textContent of the link (John Smith) and the url from the href attribute. In RDFa this is done by adding a couple of attributes to the a element. It would read like this: [[[ This book has been authored by a property=name rel=url href= http://smith.org/john;John Smith/a. ]]] Is there any way to do the same in microdata without adding a new HTML element to the markup? No, Microdata purposely keeps its data model simple by expressing property names through a single attribute. Since having @itemprop on an a always refers to the @href of the element, you must nest an additional element, such as a span, into your markup to carry the property that refers to the text content. Thanks for clarifying this, Tab. Steph. ~TJ
Re: [whatwg] adding microdata to basic links
On Wed, Aug 24, 2011 at 10:22, Stéphane Corlosquet scorlosq...@gmail.com wrote: On Wed, Aug 24, 2011 at 1:18 PM, Tab Atkins Jr. jackalm...@gmail.comwrote: On Wed, Aug 24, 2011 at 10:02 AM, Stéphane Corlosquet scorlosq...@gmail.com wrote: Starting from a basic markup like this: [[[ This book has been authored by a href=http://smith.org/john;John Smith/a. ]]] I would like to markup both the textContent of the link (John Smith) and the url from the href attribute. In RDFa this is done by adding a couple of attributes to the a element. It would read like this: [[[ This book has been authored by a property=name rel=url href= http://smith.org/john;John Smith/a. ]]] Stéphane, this looks like an incomplete example - how does a parser know where the object that has name and url property starts and ends, is it the whole page? The nearest common ancestor element? AKA how does a parser determine the scope of the object? And are name and url intended to be page-specific properties, or do they belong to a theoretical shared vocabulary? Could you provide a complete RDFa example of what you're attempting to accomplish? Is there any way to do the same in microdata without adding a new HTML element to the markup? No, Microdata purposely keeps its data model simple by expressing property names through a single attribute. One person (parser developer's) simple, is another person's (web author/designer/publisher) odd new strange way, and thus far from simple. It's a trade-off rather than being simple in any absolute terms. Since having @itemprop on an a always refers to the @href of the element, you must nest an additional element, such as a span, into your markup to carry the property that refers to the text content. This does seem to be a (fairly common) case where microdata requires additional markup (another element) whereas both microformats (e.g. hCard) and microdata (through the perhaps questionable overloading of 'rel') do not. Stéphane, if you could provide a complete RDFa example of the content example you gave, I'd like to see what Tab (or anyone else) sees as an ideal way to mark it up with microdata instead for comparison purposes. Thanks, Tantek -- http://tantek.com/ - I made an HTML5 tutorial! http://tantek.com/html5
Re: [whatwg] adding microdata to basic links
Hi Tantek, On Wed, Aug 24, 2011 at 1:30 PM, Tantek Çelik tan...@cs.stanford.eduwrote: On Wed, Aug 24, 2011 at 10:22, Stéphane Corlosquet scorlosq...@gmail.com wrote: On Wed, Aug 24, 2011 at 1:18 PM, Tab Atkins Jr. jackalm...@gmail.com wrote: On Wed, Aug 24, 2011 at 10:02 AM, Stéphane Corlosquet scorlosq...@gmail.com wrote: Starting from a basic markup like this: [[[ This book has been authored by a href=http://smith.org/john;John Smith/a. ]]] I would like to markup both the textContent of the link (John Smith) and the url from the href attribute. In RDFa this is done by adding a couple of attributes to the a element. It would read like this: [[[ This book has been authored by a property=name rel=url href= http://smith.org/john;John Smith/a. ]]] Stéphane, this looks like an incomplete example - how does a parser know where the object that has name and url property starts and ends, is it the whole page? The nearest common ancestor element? AKA how does a parser determine the scope of the object? And are name and url intended to be page-specific properties, or do they belong to a theoretical shared vocabulary? Could you provide a complete RDFa example of what you're attempting to accomplish? I purposely left out the context markup here as I was more interested in the a element. But here is the RDFa markup. I'm not overloading it with other attribute such as the title of the book for keeping the example small. [[[ p This book has been authored by span vocab=http://schema.org/; typeof=Person a property=name rel=url href=http://smith.org/john;John Smith/a /span /p ]]] Is there any way to do the same in microdata without adding a new HTML element to the markup? No, Microdata purposely keeps its data model simple by expressing property names through a single attribute. One person (parser developer's) simple, is another person's (web author/designer/publisher) odd new strange way, and thus far from simple. It's a trade-off rather than being simple in any absolute terms. Since having @itemprop on an a always refers to the @href of the element, you must nest an additional element, such as a span, into your markup to carry the property that refers to the text content. This does seem to be a (fairly common) case where microdata requires additional markup (another element) whereas both microformats (e.g. hCard) and microdata (through the perhaps questionable overloading of 'rel') do not. I think you meant RDFa here. Steph. Stéphane, if you could provide a complete RDFa example of the content example you gave, I'd like to see what Tab (or anyone else) sees as an ideal way to mark it up with microdata instead for comparison purposes. Thanks, Tantek -- http://tantek.com/ - I made an HTML5 tutorial! http://tantek.com/html5
Re: [whatwg] adding microdata to basic links
Tantek asked: Could you provide a complete RDFa example of what you're attempting to accomplish? Stéphane replied: p This book has been authored by span vocab=http://schema.org/; typeof=Person a property=name rel=url href=http://smith.org/john;John Smith/a /span /p This could be represented in Microdata without an extra element: p itemscope This book has been authored by span itemprop=name a itemprop=url href=http://smith.org/john;John Smith/a /span /p Or in the hCard microformat with even less markup: p class=vcard This book has been authored by a class=fn url href=http://smith.org/john;John Smith/a /p HTH, Ted
Re: [whatwg] adding microdata to basic links
Hi Ted, On Wed, Aug 24, 2011 at 3:53 PM, Edward O'Connor eocon...@apple.com wrote: Tantek asked: Could you provide a complete RDFa example of what you're attempting to accomplish? Stéphane replied: p This book has been authored by span vocab=http://schema.org/; typeof=Person a property=name rel=url href=http://smith.org/john;John Smith/a /span /p This could be represented in Microdata without an extra element: p itemscope This book has been authored by span itemprop=name a itemprop=url href=http://smith.org/john;John Smith/a /span /p Not the same. The span element in my example was used to assert a type, which you repurposed for the name property. So no matter what, you need to wrap the a element or its textContent with an extra span element for the name property, e.g. p itemscope This book has been authored by span itemscope itemtype=http://schema.org/Person; span itemprop=name a itemprop=url href=http://smith.org/john;John Smith/a /span /span /p Steph. Or in the hCard microformat with even less markup: p class=vcard This book has been authored by a class=fn url href=http://smith.org/john;John Smith/a /p HTH, Ted
Re: [whatwg] adding microdata to basic links
I wrote: This could be represented in Microdata without an extra element: p itemscope This book has been authored by span itemprop=name a itemprop=url href=http://smith.org/john;John Smith/a /span /p Not the same. The span element in my example was used to assert a type Ahh, sorry, I dropped the type. But I did so by moving the itemscope to the p from the span. You can dangle the type off of p and avoid the extra element like so: p itemscope itemtype=http://schema.org/Person; This book has been authored by span itemprop=name a itemprop=url href=http://smith.org/john;John Smith/a /span /p
