> If the reported value is incorrect, the A/V synchronization won't be possible. (Perhaps chcunningham@ can say more about this use case.)
The typical video player design drives the clock with audio. In short, you send buffers of audio to the OS, use the latency value to know when those buffers actually reach the user's ears, and choose a video frame accordingly. Apps using WebCodecs will design apps this way (example here <https://github.com/chcunningham/wc-talk#simple_video_playerhtml>). As Mike notes, the latency value can vary by quite a lot depending on the device (huge differences for bluetooth vs wired), so it's critical to have the platform tell you when this changes. In terms of precision, we can _probably_ do some rounding like to the nearest millisecond. Higher values may lead to issues; user studies <https://en.wikipedia.org/wiki/Audio-to-video_synchronization#Recommendations> have shown sensitivity to fairly low values of drift. Aside: we may eventually desire a display latency value for similar reasons. On Thu, Nov 18, 2021 at 12:45 PM Hongchan Choi <[email protected]> wrote: > Hi Yoav, > > Could you clarify what needs to be discussed from the spec side? The > feature (and its behavior) is properly specified, and its privacy aspect is > already reviewed by W3C: > https://github.com/WebAudio/web-audio-api/issues/1498 (outputLatency) > https://github.com/WebAudio/web-audio-api/issues/1495 (general > security/privacy consideration) > > I thought the request from this thread is to devise a Chrome-specific > mitigation, not reopening the spec discussion. Please correct me if I'm > mistaken. > > Cheers, > Hongchan > > > On Thu, Nov 18, 2021 at 12:37 PM Yoav Weiss <[email protected]> > wrote: > >> Hey folks! >> >> It seems like there are some things to figure out here before shipping. >> Could y'all file a spec issue to discuss this with the broader community? >> >> On Tuesday, November 16, 2021 at 6:16:14 PM UTC+1 hongchan wrote: >> >>> Thanks for the feedback, Mike. This is really helpful! >>> >>> Mitigation strategies include adding jitter (dithering) and quantization >>>> so that the exact skew is incorrectly reported >>>> >>> >>> This bit refers to the randomization mechanism deployed by other >>> browsers, but this approach defeats the purpose of this specific API and it >>> is also argued in the spec: >>> >>> Note however that most audio systems aim for low latency, to synchronise >>>> the audio generated by WebAudio to other audio or video sources or to >>>> visual cues (for example in a game, or an audio recording or music making >>>> environment). Excessive latency decreases usability and may be an >>>> accessibility issue. >>> >>> >>> If the reported value is incorrect, the A/V synchronization won't be >>> possible. (Perhaps chcunningham@ can say more about this use case.) >>> >>> One mitigation idea: we can do some research to collect values reported >>> by major platforms/devices and devise a reasonable "bucket system" for them. >>> >>> >>> On Tue, Nov 16, 2021 at 8:14 AM Mike Taylor <[email protected]> >>> wrote: >>> >>>> I suppose it would be nice to do some testing to verify the idea that >>>> there's a small number of platform + audio device pairs. I was hoping >>>> Mozilla's >>>> hardware survey <https://data.firefox.com/dashboard/hardware> might >>>> have something on that, but it doesn't track speakers. >>>> >>>> Just from some simple testing with the following script (thanks >>>> dmcardle@ for the snippet) in Firefox on macOS by 3 people, we have >>>> very different values that change dramatically when you unplug a headset: >>>> >>>> ``` >>>> (function getNextSample() { >>>> console.log((new AudioContext).outputLatency); >>>> setTimeout(getNextSample, 10); >>>> })(); >>>> ``` >>>> Here's the values on my macbook, using Firefox Nightly: >>>> bose headset plugged in: 0.005895833 >>>> take headset out: 0.013083333 >>>> plug bose headset back in: 0.005875 (different, but stable) >>>> >>>> https://www.w3.org/TR/webaudio/#priv-sec mentions some mitigation >>>> strategies - do we have a plan to apply any of them? >>>> >>>> On 11/15/21 11:53 AM, Ajay Rahatekar wrote: >>>> >>>> The small number of platform + audio output device combination i.e. a >>>> large number of users with the same platform and output device (users on >>>> macOS using airpods) would likely be a natural obfuscator. Not sure if we >>>> have data about the distribution. >>>> >>>> On Mon, Nov 15, 2021 at 8:05 AM Mike Taylor <[email protected]> >>>> wrote: >>>> >>>>> On 11/15/21 10:56 AM, Hongchan Choi wrote: >>>>> >>>>> On Mon, Nov 15, 2021 at 6:49 AM Mike Taylor <[email protected]> >>>>> wrote: >>>>> >>>>>> On 11/15/21 1:47 AM, Yoav Weiss wrote: >>>>>> >>>>>> On Fri, Nov 12, 2021 at 5:33 PM 'Ajay Rahatekar' via blink-dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> TAG review status >>>>>>> >>>>>>> Completed: Web Audio API specification is W3C Recommendation. >>>>>>> >>>>>>> Risks >>>>>>> >>>>>>> There is a risk of the feature being used for fingerprinting. >>>>>>> However outputLatency is the buffer size of the platform-provided >>>>>>> audio callback, so the value is inherently platform-specific. That said, >>>>>>> the majority of the platform audio buffer size is widely known. (MacOS = >>>>>>> 128 frames, Windows = 10ms, Android = 96 frames, etc) >>>>>>> >>>>>>> This feature does not expose more than what you can query/infer >>>>>>> from the UA string. >>>>>>> >>>>>> >>>>>> Would that exposure map cleanly to UA-Platform >>>>>> <https://wicg.github.io/ua-client-hints/#sec-ch-ua-platform>, which >>>>>> is considered low-entropy and exposed by default? Or would it add more >>>>>> than >>>>>> that? >>>>>> >>>>>> /cc +Mike Taylor <[email protected]> >>>>>> >>>>>>> Looking at >>>>>> https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency, it >>>>>> states that it depends on the platform _and_ the hardware output device. >>>>>> If >>>>>> I use an app using outputLatency with speaker A, then switch to speaker >>>>>> B, >>>>>> will the outputLatency remain the same? >>>>>> >>>>> >>>>> The specification says: If the audio output device is changed the >>>>> outputLatency attribute value will be updated accordingly. >>>>> >>>>> So the answer is no. The value will change accordingly. >>>>> >>>>> If that's the case, then outputLatency can reveal more entropy than >>>>> just platform alone, right? It would be useful to know what this looks >>>>> like >>>>> in practice, or what mitigations we might be able to apply depending on >>>>> the >>>>> size of these latency differences. >>>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSofF8iKX7VXh9kgfb6E0EcJ4yfUkaJCzRW-YDsMtfkndg%40mail.gmail.com.
