Proposal based on our meeting: - Requesting to make this ENABLED_BY_DEFAULT in M112 but to have a Reverse Origin Trial available with an end date date 3 milestones out: M115.
On Tuesday, February 7, 2023 at 10:19:35 PM UTC+1 Chris Thompson wrote: > > Do we have trackable statistics on the usage of corporate policies? > > ie if nobody uses the policy in 2 milestones, can we detect that and > decide that it is not needed and delete it, or will we be as unsure as we > are now? > > It is quite common in my experience to handle removals with an enterprise > policy that is stated upfront to be temporary (i.e., the enterprise release > notes give a deadline milestone), with no metrics needed. > > On Tue, Feb 7, 2023 at 1:17 PM 'Manjesh Malavalli' via blink-dev < > blink-dev@chromium.org> wrote: > >> Hi team, >> >> Thank you for postponing the rollout to M112. This gives our customers >> some time to migrate to the newest version of the Twilio Video SDK, which >> does not depend on the removed stats. We will do an internal post-mortem >> regarding why our tests did not catch this sooner and make the necessary >> corrections on our end. >> >> Thanks, >> >> Manjesh >> >> On Tuesday, February 7, 2023 at 2:36:43 AM UTC-8 Henrik Boström wrote: >> >>> I have the same concern as Harald regarding corporate policies. Why not >>> a Deprecation Origin Trial in that case for list of users and more concrete >>> timeline? >>> >>> On Tuesday, February 7, 2023 at 9:10:40 AM UTC+1 Harald Alvestrand wrote: >>> >>>> Do we have trackable statistics on the usage of corporate policies? >>>> ie if nobody uses the policy in 2 milestones, can we detect that and >>>> decide that it is not needed and delete it, or will we be as unsure as we >>>> are now? >>>> >>>> >>>> On Mon, Feb 6, 2023 at 9:41 PM Mike Taylor <mike...@chromium.org> >>>> wrote: >>>> >>> We don't know what we don't know, but it's not hard to imagine an >>>>> in-house enterprise WebRTC application that is using "stats" or "track". >>>>> Twilio is the breakage we know about (because a developer took the time >>>>> to >>>>> report a bug). Having a policy so an app continues to work while a fix is >>>>> made is a good thing - and comes with the nice side effect of appearing >>>>> on >>>>> the Enterprise release notes, increasing awareness. >>>>> >>>>> On 2/4/23 3:28 AM, Harald Alvestrand wrote: >>>>> >>>> What's the imagined scenario in which an enterprise policy would be >>>>> useful? >>>>> >>>>> The only place I could imagine it being relevant is if there exists a >>>>> WebRTC application that is only used within a single enterprise (neither >>>>> hosting nor usage exists outside the enterprise), and that WebRTC >>>>> application depends on non-upgraded Twilio libraries. >>>>> >>>>> I don't know that we have evidence that such applications exist. >>>>> >>>>> >>>>> >>>>> On Fri, Feb 3, 2023 at 6:14 PM Mike Taylor <mike...@chromium.org> >>>>> wrote: >>>>> >>>>> I agree with Johnny that an enterprise policy would be useful, at >>>>>> least for a few milestones. >>>>>> >>>>>> On 1/30/23 5:16 AM, 'Harald Alvestrand' via blink-dev wrote: >>>>>> >>>>> I'm not sure an enterprise policy is appropriate - I see the same >>>>>> problem with sunsetting the policy as with sunsetting the stat in >>>>>> general, >>>>>> and usage of enterprise policies is (as far as I know) far more opaque >>>>>> to >>>>>> us than origin trials or Finch feature usage. >>>>>> >>>>>> >>>>>> On Mon, Jan 30, 2023 at 11:13 AM Henrik Boström <hb...@chromium.org> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, January 27, 2023 at 7:24:58 PM UTC+1 Johnny Stenback >>>>>>> wrote: >>>>>>> Is there an enterprise policy in place for this deprecation already? >>>>>>> If not, adding one seems appropriate given the challenges of rolling >>>>>>> out >>>>>>> even simple fixes in some enterprise environments. >>>>>>> >>>>>>> One does not exist at the moment but I can add one >>>>>>> <https://chromium.googlesource.com/chromium/src/+/HEAD/docs/enterprise/add_new_policy.md> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Johnny >>>>>>> >>>>>>> On Fri, Jan 27, 2023 at 5:16 AM Henrik Boström <hb...@chromium.org> >>>>>>> wrote: >>>>>>> Delaying the enabled-by-default to M112 is fine by me but I'll wait >>>>>>> for a resolution here before taking action. Currently it is >>>>>>> enabled-by-default in Canary. >>>>>>> >>>>>>> On Friday, January 27, 2023 at 12:41:23 PM UTC+1 >>>>>>> philipp...@googlemail.com wrote: >>>>>>> Am Fr., 27. Jan. 2023 um 11:49 Uhr schrieb Henrik Boström < >>>>>>> hb...@chromium.org>: >>>>>>> *Contact emails* >>>>>>> hb...@chromium.org, h...@chromium.org >>>>>>> >>>>>>> *Background* >>>>>>> I attempted to remove this feature before but had forgotten to file >>>>>>> an intent to deprecate, background here >>>>>>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/RsIktnGhHqw/> >>>>>>> . >>>>>>> >>>>>>> *Specification* >>>>>>> The getStats() API spec is here >>>>>>> <https://w3c.github.io/webrtc-stats/> and it contains all the >>>>>>> metrics. The deprecated metrics are also listed, but in the obsolete >>>>>>> section >>>>>>> <https://w3c.github.io/webrtc-stats/#obsolete-rtcmediastreamtrackstats-members>. >>>>>>> >>>>>>> There's an open issue to remove obsolete metrics from the spec as soon >>>>>>> as >>>>>>> they are unshipped from modern browsers. This is considered a blocker >>>>>>> for >>>>>>> the document to reach Proposed Recommendation status. >>>>>>> >>>>>>> *Summary* >>>>>>> WebRTC is a set of JavaScript APIs (spec >>>>>>> <https://w3c.github.io/webrtc-pc/>) that allow real-time >>>>>>> communication between browsers. For the relevant metrics being removed, >>>>>>> we're only talking about the WebRTC use case that is sending or >>>>>>> receiving >>>>>>> audio or video (typically Video Conferencing use cases), not the data >>>>>>> channel use cases that is a popular WebRTC use case, since data channel >>>>>>> only use cases would never have any tracks/streams. >>>>>>> >>>>>>> RTCPeerConnection.getStats() returns a map of string-to-objects, >>>>>>> where each object is one of the dictionaries defined in the stats spec. >>>>>>> The >>>>>>> reason an app calls getStats() is mostly to report quality metrics >>>>>>> (send >>>>>>> and receive resolutions, bitrates, glitches, video QP, etc) which can >>>>>>> be >>>>>>> important for A/B experimentation. It can also be used in a way that >>>>>>> impacts app logic or even UX inside the app. Most common use case I can >>>>>>> think of: poll getStats() at 10 Hz and render volume bars for each >>>>>>> participant based on volume levels from stats objects. >>>>>>> >>>>>>> The deprecation in question is to remove some stats objects that >>>>>>> were made obsolete several years ago: all metrics on the "track" >>>>>>> dictionary >>>>>>> have been moved to non-obsolete objects ("inbound-rtp", "outbound-rtp", >>>>>>> "media-source"). Reasons for wanting to deprecate include: >>>>>>> >>>>>>> - Spec-compliance: needed for browser implementations to align >>>>>>> and for the spec to become Proposed Recommendation. >>>>>>> - Web compat: Firefox never implement "track" or "steam" >>>>>>> >>>>>>> <https://wpt.fyi/results/webrtc-stats/supported-stats.https.html?label=experimental&label=master&aligned> >>>>>>> due >>>>>>> to them being obsolete. >>>>>>> - Performance: the duplicated metrics make up ~40% of the stats >>>>>>> report size, which can be a significant number of bytes in larger >>>>>>> meetings >>>>>>> and it is common for apps to poll getStats() 10 times per second. >>>>>>> - Tech debt: unblock removal of 1400 LOC. >>>>>>> >>>>>>> In the meantime, the obsolete metrics is duplicated in several >>>>>>> places of the stats report. >>>>>>> >>>>>>> *Risks* >>>>>>> *- Impossible to properly measure usage* >>>>>>> Because stats objects are exposed as JavaScript dictionaries, and >>>>>>> because apps have to iterate all objects of the stats report in order >>>>>>> to >>>>>>> find the ones they are interested in or if they just dump all the data >>>>>>> without filtering, there is no way to measure how big the dependency is >>>>>>> on >>>>>>> track in the real world. >>>>>>> >>>>>>> While we lack use counters, we have some positive signs: >>>>>>> >>>>>>> - Because Firefox does not have "track" or "stream" stats, any >>>>>>> app that can run on Firefox already exercises the paths of these not >>>>>>> existing. >>>>>>> >>>>>>> >>>>>>> - An experiment to "unship deprecated metrics" has been running >>>>>>> at 50% Canary since October >>>>>>> >>>>>>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/RsIktnGhHqw/m/3iqjODsMBwAJ>, >>>>>>> >>>>>>> giving developers testing Canary a heads-up. Nobody complained until >>>>>>> the >>>>>>> experiment reached Stable. >>>>>>> - We got to 50% Stable in M109 and while we're in the process of >>>>>>> rolling it back now due to breaking twilio-video.js >>>>>>> <https://github.com/twilio/twilio-video.js/issues/1968>, it's >>>>>>> interesting to note that this is the only breakage we are aware of >>>>>>> (that >>>>>>> does not mean there aren't more breakages, but I believe this at >>>>>>> least says >>>>>>> something about the severity). >>>>>>> >>>>>>> *- Selenium et al typically starts browsers from fresh profiles and >>>>>>> hence does not know the finch trial seed* >>>>>>> The most likely explanation for breakage is not testing Canary or >>>>>>> test environments not having access to Finch experiments. This makes >>>>>>> the >>>>>>> behavior on Stable a surprise. >>>>>>> >>>>>>> *- To have a Reverse Origin Trial or not to have a Reverse Origin >>>>>>> Trial?* >>>>>>> Migrating should require so few lines of code (look for stats.type >>>>>>> == 'inbound-rtp' instead of stats.type == 'track', for example) that it >>>>>>> seems to be a bigger hurdle for a developer to enroll in the trial than >>>>>>> to >>>>>>> simply fix their code. >>>>>>> >>>>>>> *- Compatiblity risk* >>>>>>> There is one particular metric out of all metrics that, if you run >>>>>>> Safari, does not exist on "inbound-rtp" yet. This can be a problem, but >>>>>>> again is probably not a big problem because this particular metric was >>>>>>> never implemented on Firefox so apps already need to survive without >>>>>>> it, >>>>>>> and it is very easy to write a fallback path for the Safari case: >>>>>>> >>>>>>> let trackIdentifer = null; // In Firefox this will never be set >>>>>>> regardless. >>>>>>> if (inboundRtp.trackIdentifier) { >>>>>>> // Spec-compliant browser. >>>>>>> trackIdentifier = inboundRtp.trackIdentifier; >>>>>>> } else if (inboundRtp.trackId) { >>>>>>> // Fallback-path for Safari or 1+ year old Chromium browsers. >>>>>>> trackIdentifier = report.get(inboundRtp.trackId).trackIdentifier; >>>>>>> } >>>>>>> >>>>>>> *Proposal* >>>>>>> Rollback to 0% Stable but keep the "unship deprecated" experiment at >>>>>>> 50% Canary/Beta. Wait for Twilio to fix their issue and do another >>>>>>> rollout >>>>>>> attempt. Keep a slower rollout pace next time. >>>>>>> >>>>>>> I see limited amount of value in a Reverse Origin Trial since it >>>>>>> appears to be more effort to register to the trial than to fix the >>>>>>> issue, >>>>>>> if you are affected. >>>>>>> >>>>>>> I do prefer to have the feature enabled-by-default in M111+ and >>>>>>> overwrite that default via Finch rather than the other way around as to >>>>>>> not >>>>>>> "turn off the fire alarm" for non-Finch testing environments. I realize >>>>>>> that is not perfect (what if you run in a non-Finch environment) but it >>>>>>> would reduce overall risk. >>>>>>> >>>>>>> Thank you Henrik. I agree with one suggestion: only do default-off >>>>>>> in M112+ (which is branching so you would just need to revert this >>>>>>> commit >>>>>>> <https://chromiumdash.appspot.com/commit/3a4d52f365df03413a856ea20366b36e8fb8ea0b> >>>>>>> on >>>>>>> the M111 branch). >>>>>>> This gives developers another month to update (which itself should >>>>>>> be quick) and then rolling it out to their customers and users (which >>>>>>> takes >>>>>>> time). >>>>>>> >>>>>>> I hope that the 50% rollout caused enough incidents (even though you >>>>>>> may never hear about some of them) to get the fixes in ASAP. >>>>>>> >>>>>>> cheers >>>>>>> >>>>>>> Philipp >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink- >>>>>>> dev/5ecf1ea6-c16c-464a-b529-439e05e4feedn%40chromium.org >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5ecf1ea6-c16c-464a-b529-439e05e4feedn%40chromium.org?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink- >>>>>>> dev/6a0b5bb9-addd-4a56-b053-1429bbaabe2dn%40chromium.org >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6a0b5bb9-addd-4a56-b053-1429bbaabe2dn%40chromium.org?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "blink-dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVG_P0ji6Z-n0DoCua4nzJZN7S48o23%3DUpx_ELerqphfUg%40mail.gmail.com >>>>>> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVG_P0ji6Z-n0DoCua4nzJZN7S48o23%3DUpx_ELerqphfUg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to blink-dev+unsubscr...@chromium.org. >> > To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/69b1146f-d759-45f6-8276-3ede63f99c22n%40chromium.org >> >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/69b1146f-d759-45f6-8276-3ede63f99c22n%40chromium.org?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/24712232-d694-4732-bf00-537f97b07a6cn%40chromium.org.
