Re: [whatwg] Removing mediagroup/MediaController from HTML

[Philip Jägenstedt]

On Tue, Nov 3, 2015 at 4:20 PM, David Singer  wrote:
>
>> On Oct 3, 2015, at 13:39 , Silvia Pfeiffer 
wrote:
>>
>> On Fri, Oct 2, 2015 at 10:27 AM, Domenic Denicola  wrote:
>>> From: whatwg [mailto:whatwg-boun...@lists.whatwg.org] On Behalf Of
>>>
 is removal really the right thing to do, given that we have an
 implementation?
>>>
>>> I agree this is a problematic question. I opened
https://github.com/whatwg/html/issues/209 for the more general issue but am
happy to have the discussion here since that hasn't gotten much replies. Do
check out the examples listed there though. E.g. Blink is in similar
situations with  and HTML imports.
>>>
>>> The web seems to end up with a lot of APIs like this, where the spec
ends up just being documentation for a single-vendor implementation. I
don't really know what to do in these cases. If our goal in writing these
specs is to produce an interoperable web platform, then such features seem
like they shouldn't be part of the platform.
>>
>>
>> There is also a question about the why of the current state: is it
>> just a single-vendor implementation because nobody at the other
>> vendors has gotten around to implementing it or is it because they
>> fundamentally object to implementing it. If there are objections, then
>> it's reasonable to consider removing the feature. Otherwise, it would
>> be premature to remove it IMHO.
>
> Yes.  It wasn’t even our proposal (see <
https://lists.w3.org/Archives/Public/public-html/2011Mar/0436.html>) and we
feel it answers some important cases that we can’t otherwise answer.  Some
insights from others would be welcome.

My main concern with the API, and the reason I unshipped it in Blink, is
the apparent difficulty in implementing it really well, to actually get the
sync sample-accurate at the media framework level. I understand that it's
not really *that* hard in principle, but still enough work that it wasn't
done for Safari, or Chromium. Some frameworks seem to make it fairly
straightforward, I wouldn't be surprised if QuickTime has supported it
since the 90's. When I looked into implementing it using GStreamer for
Presto, it seemed like it was a matter of using a single pipeline with a
single audio output node and thus a single clock. What seems quite tricky
to me is to dynamically split or merge pipelines as media elements join or
leave a media controller. I never really looked much deeper than this,
maybe it's quite doable.

Another more recent concern that I have is that media controller builds on
top of an already quite high-level abstraction that is HTMLMediaElement. In
the interests of a layered and (somewhat) rational platform, I wonder if it
wouldn't be more sensible to crack open HTMLMediaElement, exposing the
primitives all the way down clocks and audio output devices, so that one
could create and connect demuxers and decoders to the same driving clock in
a way that's more similar to how you'd to it using the underlying
frameworks. Of course this isn't even half-baked, and it would be a
many-year project to bring sanity to the media stack (media elements + MSE
+ EME + Web Audio + WebRTC) of the web.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:43 AM, Eric Carlson  wrote:
>
>> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
>>
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith > > wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
 On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
 wrote:
> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>
>>>
>>> [...]
 I've filed a spec issue to make it so:
 https://github.com/whatwg/html/issues/262

 If there's any implementor interest in pitch control that goes beyond
 (independently) or that, please file a separate issue.

>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>   I am actually quite ambivalent about this feature - it is currently broken 
> on OSX and it has never been implemented on iOS, but as far as I can see we 
> haven’t received a single bug report about this.

Ah. I removed webkitPreservesPitched entirely from Blink because it
wasn't implemented on any platforms, maybe you could do the same on
iOS?

More importantly, is it possible to support this on iOS, and is it
likely to bubble to the top of your priorities any time soon if it
were added to the spec? I don't think it's high priority for anyone
working on Chromium, but unless Safari and Firefox are ready to drop
their prefixed APIs entirely it still seems like a good idea to get
what little interop we can by standardizing.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith 
wrote:
> On 11/12/15, Philip Jägenstedt  wrote:
>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
>> wrote:
>>>
>>> On 10/19/15, Philip Jägenstedt  wrote:
>>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> > wrote:
>>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>>> >> wrote:
>>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>> >>>
>>>
>>> [...]
>>> > I've filed a spec issue to make it so:
>>> > https://github.com/whatwg/html/issues/262
>>> >
>>> > If there's any implementor interest in pitch control that goes beyond
>>> > (independently) or that, please file a separate issue.
>>> >
>>>
>>> They won't.
>>>
>>> You can hold the standard of "they need to come here and post up
>>> cogent arguments in favor of feature X", but it ain't gonna happen
>>> that way.
>>>
>>> There just isn't a whole lot of money in music education. How many
>>> music education companies are W3C members?
>>>
>>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>>> other companies the post here, small music education operations like
>>> artistworks, jammit, licklibrary, etc are more about their domain —
>>> "music" — than they are about technology.
>>>
>>> Major music education websites are still using Flash; their developers
>>> are busy fixing broken links, making the login feature, database, etc
>>> work, etc. Flash is not nice but they apparently were not funded or
>>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>>> instead.
>>>
>>> Control over playbackRate has a greater value than pitch control. But
>>> those sites don't even allow the students to change the playbackRate
>>> because they're still using Flash.
>>>
>>> You won't read posts here about what students have to say about it the
>>> value of having HTML5 vs Flash, or independent control over pitch and
>>> playbackRate.
>>
>> Have you investigated whether you can achieve your use cases using the
>> Web Audio API? If it isn't possible, is there a small addition to Web
>> Audio that would solve the problem?
>>
>
> https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

I don't understand, isn't the AudioTrack API (enabling/disabling individual
audio tracks) separate a separate concern? Or are you saying that there's
no way to get the audio tracks separately into the Web Audio API in order
to process them differently? That is certainly true, and if having that
capability would allow you to solve the problem with the Web Audio API, I
think that seems like a powerful primitive to have.

>> It is unfortunately quite hard to get all browser vendors (or even
>> one) interested in implementing support for something that is only
>> expected to benefit a niche use case, but we should strive to make
>> available the primitives that would it possible to implement yourself.
>>
>
> The situation where you want hear the audio at the desired pitch,
> while simultaneously watching the video at the desired speed, to see
> it performed, and to play along. It doesn't work well when trying to
> play to a video example, such as playing an open G chord, and the
> sound from the video does not match the sound from your own guitar.
>
> What I can do now on websites that use VIDEO, such as YouTube, is
> limited to using the web inspector (console). I click "inspect
> element" to find the VIDEO element, then change its DOM properties for
> playbackRate = .95 and mozPreservesPitch = false. That'll get
> everything down a half-step (although at 95% speed).
>
> For example, watch Andy James play "Harvester of Sorrow" but hear it
> 1/2 step down:
>
> https://www.youtube.com/watch?v=vYZjC_U1VJg
>
> 1) Right click on the video "Inspect Element"
> 2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
> 3) set playbackRate: 0.95, mozPreservesPitch: 0
>
> Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
> few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
> down further.
>
> A=440 (standard tuning) is much more common. When you want to learn
> something played tuned down 1/2 step and your guitar is in standard
> tuning, it is a problem. Constantly retuning the instrument is not
> practical. Especially for guitars with locking nut systems or for
> other instruments like piano. I don't know about learning horn parts.
> Or you can buy multiple instruments and maintain different tunings for
> them.
>
> Fast passages, it's helpful to slow the speed down but keep the pitch
> as original and in some cases, you want to slow the speed down and
> adjust the pitch separately, so you can do things like learning Yngwie
> Malmsteen songs while using a guitar in standard tuning.
>
> I realize there's not much demand for it. But there seems to be
> separation of playback pitch with `preservesPitch` and 

Re: [whatwg] VIDEO and pitchAdjustment

[Eric Carlson]

> On Nov 12, 2015, at 9:34 AM, Philip Jägenstedt  wrote:
> 
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  > wrote:
>> 
>> On 10/19/15, Philip Jägenstedt  wrote:
>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>>> wrote:
 On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> From: Eric Carlson [mailto:eric.carl...@apple.com]
> 
>> 
>> [...]
>>> I've filed a spec issue to make it so:
>>> https://github.com/whatwg/html/issues/262
>>> 
>>> If there's any implementor interest in pitch control that goes beyond
>>> (independently) or that, please file a separate issue.
>>> 
>> 
>> They won't.
>> 
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>> 
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>> 
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>> 
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>> 
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>> 
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
> 
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
> 
> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
> 
  I am actually quite ambivalent about this feature - it is currently broken on 
OSX and it has never been implemented on iOS, but as far as I can see we 
haven’t received a single bug report about this.

eric

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 10/19/15, Philip Jägenstedt  wrote:
> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> wrote:
>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
>>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>>>

[...]
> I've filed a spec issue to make it so:
> https://github.com/whatwg/html/issues/262
>
> If there's any implementor interest in pitch control that goes beyond
> (independently) or that, please file a separate issue.
>

They won't.

You can hold the standard of "they need to come here and post up
cogent arguments in favor of feature X", but it ain't gonna happen
that way.

There just isn't a whole lot of money in music education. How many
music education companies are W3C members?

Unlike technology companies like Facebook, Google, Nokia, Opera, and
other companies the post here, small music education operations like
artistworks, jammit, licklibrary, etc are more about their domain —
"music" — than they are about technology.

Major music education websites are still using Flash; their developers
are busy fixing broken links, making the login feature, database, etc
work, etc. Flash is not nice but they apparently were not funded or
motivated enough by the existing HTML5 HTMLMediaElement to use it
instead.

Control over playbackRate has a greater value than pitch control. But
those sites don't even allow the students to change the playbackRate
because they're still using Flash.

You won't read posts here about what students have to say about it the
value of having HTML5 vs Flash, or independent control over pitch and
playbackRate.

-- 
Garrett / independent voice
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com

Re: [whatwg] VIDEO and pitchAdjustment

[Philip Jägenstedt]

On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith  wrote:
>
> On 10/19/15, Philip Jägenstedt  wrote:
> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
> > wrote:
> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola  wrote:
> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
> >>>
>
> [...]
> > I've filed a spec issue to make it so:
> > https://github.com/whatwg/html/issues/262
> >
> > If there's any implementor interest in pitch control that goes beyond
> > (independently) or that, please file a separate issue.
> >
>
> They won't.
>
> You can hold the standard of "they need to come here and post up
> cogent arguments in favor of feature X", but it ain't gonna happen
> that way.
>
> There just isn't a whole lot of money in music education. How many
> music education companies are W3C members?
>
> Unlike technology companies like Facebook, Google, Nokia, Opera, and
> other companies the post here, small music education operations like
> artistworks, jammit, licklibrary, etc are more about their domain —
> "music" — than they are about technology.
>
> Major music education websites are still using Flash; their developers
> are busy fixing broken links, making the login feature, database, etc
> work, etc. Flash is not nice but they apparently were not funded or
> motivated enough by the existing HTML5 HTMLMediaElement to use it
> instead.
>
> Control over playbackRate has a greater value than pitch control. But
> those sites don't even allow the students to change the playbackRate
> because they're still using Flash.
>
> You won't read posts here about what students have to say about it the
> value of having HTML5 vs Flash, or independent control over pitch and
> playbackRate.

Have you investigated whether you can achieve your use cases using the
Web Audio API? If it isn't possible, is there a small addition to Web
Audio that would solve the problem?

It is unfortunately quite hard to get all browser vendors (or even
one) interested in implementing support for something that is only
expected to benefit a niche use case, but we should strive to make
available the primitives that would it possible to implement yourself.

Philip

Re: [whatwg] VIDEO and pitchAdjustment

[Garrett Smith]

On 11/12/15, Philip Jägenstedt  wrote:
> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith 
> wrote:
>>
>> On 10/19/15, Philip Jägenstedt  wrote:
>> > On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt 
>> > wrote:
>> >> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola 
>> >> wrote:
>> >>> From: Eric Carlson [mailto:eric.carl...@apple.com]
>> >>>
>>
>> [...]
>> > I've filed a spec issue to make it so:
>> > https://github.com/whatwg/html/issues/262
>> >
>> > If there's any implementor interest in pitch control that goes beyond
>> > (independently) or that, please file a separate issue.
>> >
>>
>> They won't.
>>
>> You can hold the standard of "they need to come here and post up
>> cogent arguments in favor of feature X", but it ain't gonna happen
>> that way.
>>
>> There just isn't a whole lot of money in music education. How many
>> music education companies are W3C members?
>>
>> Unlike technology companies like Facebook, Google, Nokia, Opera, and
>> other companies the post here, small music education operations like
>> artistworks, jammit, licklibrary, etc are more about their domain —
>> "music" — than they are about technology.
>>
>> Major music education websites are still using Flash; their developers
>> are busy fixing broken links, making the login feature, database, etc
>> work, etc. Flash is not nice but they apparently were not funded or
>> motivated enough by the existing HTML5 HTMLMediaElement to use it
>> instead.
>>
>> Control over playbackRate has a greater value than pitch control. But
>> those sites don't even allow the students to change the playbackRate
>> because they're still using Flash.
>>
>> You won't read posts here about what students have to say about it the
>> value of having HTML5 vs Flash, or independent control over pitch and
>> playbackRate.
>
> Have you investigated whether you can achieve your use cases using the
> Web Audio API? If it isn't possible, is there a small addition to Web
> Audio that would solve the problem?
>

https://html.spec.whatwg.org/multipage/embedded-content.html#audiotrack

> It is unfortunately quite hard to get all browser vendors (or even
> one) interested in implementing support for something that is only
> expected to benefit a niche use case, but we should strive to make
> available the primitives that would it possible to implement yourself.
>

The situation where you want hear the audio at the desired pitch,
while simultaneously watching the video at the desired speed, to see
it performed, and to play along. It doesn't work well when trying to
play to a video example, such as playing an open G chord, and the
sound from the video does not match the sound from your own guitar.

What I can do now on websites that use VIDEO, such as YouTube, is
limited to using the web inspector (console). I click "inspect
element" to find the VIDEO element, then change its DOM properties for
playbackRate = .95 and mozPreservesPitch = false. That'll get
everything down a half-step (although at 95% speed).

For example, watch Andy James play "Harvester of Sorrow" but hear it
1/2 step down:

https://www.youtube.com/watch?v=vYZjC_U1VJg

1) Right click on the video "Inspect Element"
2) Right click on the VIDEO in the HTML tab. Choose "Show DOM Properties"
3) set playbackRate: 0.95, mozPreservesPitch: 0

Ed Van Halen, Yngwie Malmsteen, Jimi Hendrix, Zakk Wylde, to name a
few tune down 1/2 step. Others, Black Sabbath, Racer X, have tuned
down further.

A=440 (standard tuning) is much more common. When you want to learn
something played tuned down 1/2 step and your guitar is in standard
tuning, it is a problem. Constantly retuning the instrument is not
practical. Especially for guitars with locking nut systems or for
other instruments like piano. I don't know about learning horn parts.
Or you can buy multiple instruments and maintain different tunings for
them.

Fast passages, it's helpful to slow the speed down but keep the pitch
as original and in some cases, you want to slow the speed down and
adjust the pitch separately, so you can do things like learning Yngwie
Malmsteen songs while using a guitar in standard tuning.

I realize there's not much demand for it. But there seems to be
separation of playback pitch with `preservesPitch` and `playbackRate`.
It would be much better with a control to adjustm pitch, if it is not
too much work.

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com