LGTM2 to ship, with a commitment to move this part of the spec to WICG
if it gets removed from the mediacapture-extensions spec.
On Thu, Dec 14, 2023 at 11:06 AM Alex Russell
<slightly...@chromium.org> wrote:
LGTM1
On Thursday, December 14, 2023 at 8:00:41 AM UTC-8 Rick Byers wrote:
Sounds good Henrik! Yes, from our brief discussion in the API
owners meeting I believe you have support from at least 3 API
owners to proceed in this direction. It's important to us that
we engage constructively in the standards process even in
areas of differing opinion, but it's also crucial to our
process that we not let such differences in opinion and
priority be an effective veto power over what we choose to
ship in Chromium. I'd encourage you and the WebRTC group to
formalize processes for amicably agreeing to disagree on the
importance of different use cases, while still being open to
technical feedback and doing what we reasonably can to
maximize the chance that the APIs we ship can eventually be
interoperable if priorities of the other engines someday
change [*]. API owners would also be interested to hear any
other arguments for why Chromium shipping these APIs would be
bad for the web (on this list, or anywhere else). I know
there's a messy history with WebRTC in particular and services
coming to depend on Chromium-only APIs when suitable
standards-track alternatives are available in other engines.
That's IMHO definitely not a pattern we want to risk repeating.
Of course you also need Chrome privacy and security reviews,
since it's important that features like this don't create a
hole in our careful balance of side channel attack
mitigations. But I see you already have privacy approval so
hopefully security isn't too far off. You might want to wait
for a signal there before starting implementation.
Personally I'm also less concerned about interoperability
risks when it comes to metrics API. It's already the case that
our top performance metrics (Core Web Vitals) have APIs
exposed only in Chromium. There's certainly some interop risk
there, eg. of sites optimizing in engine-specific ways. But in
practice we've seen developers use these APIs mostly to make
their sites faster in ways that generally apply to all
engines. So in that case, Safari and Firefox are getting most
of the benefit of these APIs existing without having to incur
most of the cost, which seems like a fine outcome to me. Also,
I'm confident that if we eventually agree with other engines
on some better way to expose the same information, then we can
deprecate and remove any API shape we ship today and customers
can migrate over without causing user-visible breakage
(worst-case we just return dummy values on the deprecated APIs).
Rick
[*] My favorite example of this is Pointer Events where Apple
was opposed to the use case, but also had good technical
critiques of the API. We eventually (after a lot of research,
open debate, and some flip-flopping by me) shipped a version
of the API that addressed the legitimate technical concerns
without addressing Apple's objections around the use case.
Years later when a use case materialized for Apple (the Apple
Pen), they just shipped the API in a fully interoperable
fashion. That (as well as cases of the inverse where we
realize we were wrong and unship) is what we mean by the blink
process being designed for "eventual interoperability".
On Thu, Dec 14, 2023 at 4:34 AM Henrik Boström
<h...@chromium.org> wrote:
Thanks, Rick. Responses inline.
On Wednesday, December 13, 2023 at 6:39:48 PM UTC+1 Rick
Byers wrote:
I looked into the details of the standards debate on
this issue. It sounds like it's still unclear whether
the spec for this has WG support or not, right? I
certainly wouldn't want to mislead anyone as to API
maturity / likely interoperability by shipping based
on a WebRTC WG specification if there is an unresolved
process concern.
I think it's safe to say we don't have consensus on the
frame counters (exposing dropped/glitches), while capture
latency hopefully be less contentious.
That said, I think Chromium's position on the
technical debate here is pretty clear - we do believe
there's value in such stats APIs, even IF they can
only represent browser bugs (it's why we ship a crash
reporting API
<https://wicg.github.io/crash-reporting/>, which has
been similarly controversial with Mozilla and Apple).
We disagree that "there's nothing web developers can
do about it". Maybe that's true for Apple and Mozilla,
but for Google we rely critically on developer
feedback through both open (crbug.com
<http://crbug.com>) and private (Google partnerships)
channels and we want to make it as easy as possible
for developers to report an issue they're seeing to us
in a way that's actionable. Our privacy policy limits
Chrome's visibility into what's going on in the wild.
So usually we find this requires a mix of both
site-acquired telemetry and browser-required
telemetry, and find the two can often complement each
other nicely.
Henrik, my advice if the WG doesn't have consensus for
this API is to move it to some incubation venue (like
a WICG group). You clearly have a community of web
developers who want it, so it's probably more
productive to focus standards energy among allies who
share an interest for the use case, right? If you're
willing to promptly move this to a WICG spec in the
event the WG asks to remove it from their spec, then I
don't think this debate changes anything from a blink
API owner's perspective so I'm OK treating it as
non-blocking. A subset of API owners met today
(Daniel, Mike Taylor, Philip and I), and agreed with
this stance. WDYT?
Me moving this to a WICG spec in the event that the WG
asks to remove them from mediacapture-extensions sounds
good to me.
If me and/or the WebRTC Audio Team has access to a WICG
spec for these things, that may also give us a venue for,
in the future, exploring /playout/ glitch metrics in a
more enthusiastic setting, which is in the early stages of
discussions internally.
In terms of testing, normally we ask to see the tests
land before approving an I2S. Any reason we shouldn't
wait for that here?
Given the "controversy" around the glitch metrics, the
WebRTC Audio Team wanted to get some Blink owners signals
before they spend the engineering efforts to implement
this (including WPTs).
But if Blink owners also see the value in these metrics
and we have a plan (= move to WICG in the event that they
are removed) I see no reason not to ask them to start
implementation today.
For reference, see the WPTs we added for the video track
stats here
<https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/mediacapture-extensions/MediaStreamTrack-video-stats.https.html>.
The audio track stats WPTs should be similar and also
complemented by C++ unit tests in lower layers.
Thanks,
Rick
On Fri, Dec 8, 2023 at 11:53 AM Henrik Boström
<h...@chromium.org> wrote:
Contact emails
h...@chromium.org, o...@chromium.org
<mailto:o...@chromium.org>, h...@chromium.org
Specification
https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface
<https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface>
Summary
The MediaStreamTrack Statistics API, or
`track.stats`, has already shipped for video
tracks: see previous I2S here
<https://groups.google.com/a/chromium.org/g/blink-dev/c/ttzYv-30gY4/m/2FvJpxqMGQAJ>.
This is the same API but for audio tracks, also
motivated by the app's desire to measure capture
quality. This is very important for conferencing
websites such as Google Meet, Microsoft Teams,
Zoom, Goto Meetings, etc. All of which has
expressed an interest in the audio portion of this
API.
The API is only available for getUserMedia()
sourced audio tracks, i.e. microphone, so the API
is behind a user prompt and only available during
capture.
The new interface we want to ship is
MediaStreamTrackAudioStats
<https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface>
which allow measuring two things from the audio
capture pipeline:
1. The number of audio frames, including if any
audio frames are dropped by the device, OS or User
Agent. This allows measuring glitches in captured
audio.
2. The input latency, such as due to buffering or
other delays in delivering the audio frames to the
track's sinks.
Blink component
Blink>GetUserMedia
<https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EGetUserMedia>
Risks
Interoperability and Compatibility
Because the API provides /statistics/about capture
quality, rather than providing capture
/functionality, /the interop/compatibility risk is
small.
/Gecko/: Standards position issue
<https://github.com/mozilla/standards-positions/issues/935>
/WebKit/: Standards position issue
<https://github.com/WebKit/standards-positions/issues/260>
Standardization
While the audio stats API is written by the W3C
WebRTC Working Group and track statistics overall
is not controversial, there is an ongoing
disagreement with Mozilla about whether or not
dropped frames (= totalFrames - deliveredFrames)
should be exposed to the web in the audio case.
The disagreement is tracked by this issue
<https://github.com/w3c/mediacapture-extensions/issues/129>.
Our need for this metric has been discussed at
Virtual Interims, see recording with 10:39
timestamp
<https://www.youtube.com/watch?v=xJMXnf3Qwh8&t=639s> but
no consensus could be reached (rough consensus was
reached between everyone except Mozilla). Youenn
(Apple) has shown that frames can be dropped due
to Bluetooth connection
<https://github.com/w3c/mediacapture-extensions/issues/129#issuecomment-1822624904>
and
is not just "measuring browser bugs".
/Web developers/: Positive
- The I2P
<https://groups.google.com/a/chromium.org/g/blink-dev/c/vUbD_psbPL8/m/wqq3kmZFBwAJ>
shows
support from Teams, Zoom and GoTo meetings.
- In the spec issue
<https://github.com/w3c/mediacapture-extensions/issues/129>
regarding the disagreement, more developer support
is expressed (e.g. alfredh from Nvidia and
steely-glint).
WebView application risks
None
Will this feature be supported on all six Blink
platforms (Windows, Mac, Linux, ChromeOS, Android,
and Android WebView)?
Yes
Is this feature fully tested by web-platform-tests
<https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md>?
Yes and WPTs will be written as part of
implementing this, however unit tests will also be
needed to verify accuracy of metrics on a lower
level. WPTs will be more general like "frame
counters increase over time during capture" and
that run-to-completion semantics are preserved.
Link to entry on the Chrome Platform Status
https://chromestatus.com/feature/5141112910249984
<https://chromestatus.com/feature/5141112910249984>
Links to previous Intent discussions
Intent to prototype:
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org>
--
You received this message because you are
subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
blink-dev+unsubscr...@chromium.org
<mailto:blink-dev+unsubscr...@chromium.org>.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to blink-dev+unsubscr...@chromium.org.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-HbWWNeZ4jyRr38WR75FtxOA2DUhkwrHH0-OZnjkq9cg%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-HbWWNeZ4jyRr38WR75FtxOA2DUhkwrHH0-OZnjkq9cg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
