Thanks for the clarification, Yoav! I hope the mitigation approach in the previous comment makes sense from the privacy perspective, but it will hurt the usability/value of this API significantly. For the privacy-focused discussion, should I use other venues rather than this I2S thread?
Cheers, Hongchan On Thu, Nov 18, 2021 at 10:35 PM Yoav Weiss <[email protected]> wrote: > Apologies, I misread the discussion. A spec issue is indeed not needed, as > this is already covered. > > From my perspective, since this exposes active entropy, the fact that it > exposes more data than the (passively exposed) platform is not prohibitive > on its own. > At the same time, it'd be good to work with privacy-focused folks to make > sure we expose as little information as needed to maintain the API's > functionality, but not more. > > On Fri, Nov 19, 2021 at 4:57 AM Chris Cunningham < > [email protected]> wrote: > >> > If the reported value is incorrect, the A/V synchronization won't be >> possible. (Perhaps chcunningham@ can say more about this use case.) >> >> The typical video player design drives the clock with audio. In short, >> you send buffers of audio to the OS, use the latency value to know when >> those buffers actually reach the user's ears, and choose a video frame >> accordingly. Apps using WebCodecs will design apps this way (example here >> <https://github.com/chcunningham/wc-talk#simple_video_playerhtml>). >> >> As Mike notes, the latency value can vary by quite a lot depending on the >> device (huge differences for bluetooth vs wired), so it's critical to have >> the platform tell you when this changes. In terms of precision, we can >> _probably_ do some rounding like to the nearest millisecond. Higher values >> may lead to issues; user studies >> <https://en.wikipedia.org/wiki/Audio-to-video_synchronization#Recommendations> >> have shown sensitivity to fairly low values of drift. >> >> Aside: we may eventually desire a display latency value for similar >> reasons. >> >> On Thu, Nov 18, 2021 at 12:45 PM Hongchan Choi <[email protected]> >> wrote: >> >>> Hi Yoav, >>> >>> Could you clarify what needs to be discussed from the spec side? The >>> feature (and its behavior) is properly specified, and its privacy aspect is >>> already reviewed by W3C: >>> https://github.com/WebAudio/web-audio-api/issues/1498 (outputLatency) >>> https://github.com/WebAudio/web-audio-api/issues/1495 (general >>> security/privacy consideration) >>> >>> I thought the request from this thread is to devise a Chrome-specific >>> mitigation, not reopening the spec discussion. Please correct me if I'm >>> mistaken. >>> >>> Cheers, >>> Hongchan >>> >>> >>> On Thu, Nov 18, 2021 at 12:37 PM Yoav Weiss <[email protected]> >>> wrote: >>> >>>> Hey folks! >>>> >>>> It seems like there are some things to figure out here before shipping. >>>> Could y'all file a spec issue to discuss this with the broader community? >>>> >>>> On Tuesday, November 16, 2021 at 6:16:14 PM UTC+1 hongchan wrote: >>>> >>>>> Thanks for the feedback, Mike. This is really helpful! >>>>> >>>>> Mitigation strategies include adding jitter (dithering) and >>>>>> quantization so that the exact skew is incorrectly reported >>>>>> >>>>> >>>>> This bit refers to the randomization mechanism deployed by other >>>>> browsers, but this approach defeats the purpose of this specific API and >>>>> it >>>>> is also argued in the spec: >>>>> >>>>> Note however that most audio systems aim for low latency, to >>>>>> synchronise the audio generated by WebAudio to other audio or video >>>>>> sources >>>>>> or to visual cues (for example in a game, or an audio recording or music >>>>>> making environment). Excessive latency decreases usability and may be an >>>>>> accessibility issue. >>>>> >>>>> >>>>> If the reported value is incorrect, the A/V synchronization won't be >>>>> possible. (Perhaps chcunningham@ can say more about this use case.) >>>>> >>>>> One mitigation idea: we can do some research to collect values >>>>> reported by major platforms/devices and devise a reasonable "bucket >>>>> system" >>>>> for them. >>>>> >>>>> >>>>> On Tue, Nov 16, 2021 at 8:14 AM Mike Taylor <[email protected]> >>>>> wrote: >>>>> >>>>>> I suppose it would be nice to do some testing to verify the idea that >>>>>> there's a small number of platform + audio device pairs. I was hoping >>>>>> Mozilla's >>>>>> hardware survey <https://data.firefox.com/dashboard/hardware> might >>>>>> have something on that, but it doesn't track speakers. >>>>>> >>>>>> Just from some simple testing with the following script (thanks >>>>>> dmcardle@ for the snippet) in Firefox on macOS by 3 people, we have >>>>>> very different values that change dramatically when you unplug a headset: >>>>>> >>>>>> ``` >>>>>> (function getNextSample() { >>>>>> console.log((new AudioContext).outputLatency); >>>>>> setTimeout(getNextSample, 10); >>>>>> })(); >>>>>> ``` >>>>>> Here's the values on my macbook, using Firefox Nightly: >>>>>> bose headset plugged in: 0.005895833 >>>>>> take headset out: 0.013083333 >>>>>> plug bose headset back in: 0.005875 (different, but stable) >>>>>> >>>>>> https://www.w3.org/TR/webaudio/#priv-sec mentions some mitigation >>>>>> strategies - do we have a plan to apply any of them? >>>>>> >>>>>> On 11/15/21 11:53 AM, Ajay Rahatekar wrote: >>>>>> >>>>>> The small number of platform + audio output device combination i.e. a >>>>>> large number of users with the same platform and output device (users on >>>>>> macOS using airpods) would likely be a natural obfuscator. Not sure if we >>>>>> have data about the distribution. >>>>>> >>>>>> On Mon, Nov 15, 2021 at 8:05 AM Mike Taylor <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On 11/15/21 10:56 AM, Hongchan Choi wrote: >>>>>>> >>>>>>> On Mon, Nov 15, 2021 at 6:49 AM Mike Taylor <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> On 11/15/21 1:47 AM, Yoav Weiss wrote: >>>>>>>> >>>>>>>> On Fri, Nov 12, 2021 at 5:33 PM 'Ajay Rahatekar' via blink-dev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> TAG review status >>>>>>>>> >>>>>>>>> Completed: Web Audio API specification is W3C Recommendation. >>>>>>>>> >>>>>>>>> Risks >>>>>>>>> >>>>>>>>> There is a risk of the feature being used for fingerprinting. >>>>>>>>> However outputLatency is the buffer size of the platform-provided >>>>>>>>> audio callback, so the value is inherently platform-specific. That >>>>>>>>> said, >>>>>>>>> the majority of the platform audio buffer size is widely known. >>>>>>>>> (MacOS = >>>>>>>>> 128 frames, Windows = 10ms, Android = 96 frames, etc) >>>>>>>>> >>>>>>>>> This feature does not expose more than what you can query/infer >>>>>>>>> from the UA string. >>>>>>>>> >>>>>>>> >>>>>>>> Would that exposure map cleanly to UA-Platform >>>>>>>> <https://wicg.github.io/ua-client-hints/#sec-ch-ua-platform>, >>>>>>>> which is considered low-entropy and exposed by default? Or would it add >>>>>>>> more than that? >>>>>>>> >>>>>>>> /cc +Mike Taylor <[email protected]> >>>>>>>> >>>>>>>>> Looking at >>>>>>>> https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency, it >>>>>>>> states that it depends on the platform _and_ the hardware output >>>>>>>> device. If >>>>>>>> I use an app using outputLatency with speaker A, then switch to >>>>>>>> speaker B, >>>>>>>> will the outputLatency remain the same? >>>>>>>> >>>>>>> >>>>>>> The specification says: If the audio output device is changed the >>>>>>> outputLatency attribute value will be updated accordingly. >>>>>>> >>>>>>> So the answer is no. The value will change accordingly. >>>>>>> >>>>>>> If that's the case, then outputLatency can reveal more entropy than >>>>>>> just platform alone, right? It would be useful to know what this looks >>>>>>> like >>>>>>> in practice, or what mitigations we might be able to apply depending on >>>>>>> the >>>>>>> size of these latency differences. >>>>>>> >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGJqXNtZU6kx4ofa73b7yKMHURwdzDWSE_WLfVMH5n2TghzFSg%40mail.gmail.com.
