Hey all!
Looks like I'm jumping on this train a little bit late, but still probably I’m not the last person on earth that is in the field of the web and only recently stumbled this change. After reading all this thread carefully I still feel that not everything was resolved so I’m posting it. Some of the reason that precise device detection is applied is to serve the best content for the users. Something that was mentioned in the thread previously. There’re multiple reasons for it that I see: 1. To provide the best UX based on the device – so mobile gets mobile web sites, desktop gets desktop experience and tablet is something in the gray zone – that’s where the use of the exact model is required, and based on device size, CPU capabilities and other properties the correct UI is provided. 2. Another requirement is to get full version of the browser and preferably the OS versions. As an example, if you work with video streaming, HLS, WebRTC and etc. they tend to break pretty often and could be fixed or broke again even between minor versions. And sometimes OS itself is a reason for it. So, the combination of these is required to allow/disallow some features and apply workarounds. 3. BI for detecting anomalies in specific version. You usually don’t need user’s fingerprint if the user is registered and logged in. But to be able to support the above 2 points and make other per device decisions the shortened UA is not enough. Experimenting lately with the UA-CH implementation for the websites, mostly the Critical-CH as high entropy CH are required to make the above decision on the “first hit”. The conclusion is that the proposed Critical-CH change is far from ideal, even harmful. The specs basically tell: 1. Browser sends a request to the server with a low entropy CH, e.g. “Sec-CH-UA-Mobile”. 2. Server sends in respond Critical-CH with list of high entropy CH, together with a full web page. 3. Browser detects new high entropy CH that are required and makes another request now with required CH 4. Server sends back a page that is adapted to the full UA-CH You all probably know the above, and some problems were raised already in this thread but I’ll mention them again as some weren’t answered: 1. Two requests – doubles the time till user receives the content – and there’re multiple researches that show how badly the load time affects the conversion. 2. The server renders the full page on first request and it’s being omitted: More time on meaningless respond sending, more traffic, more time on server creating the response – think of server going to DB – useless load on all layers. More power consumption… 3. How the server knows that the client will send a second request or not? Maybe the browser doesn’t support at all client hints then the server needs to prepare the whole page and not optimize internal BL. The other case is that browser supports CH but won’t send them anyway (if in the future user will opt out from some CH, if it will be available, or the privacy budget is empty). 4. BL should be rewritten as now user makes sometimes 2 requests. The alternative solution that was discussed by developers is to use redirects that are sent from the server for more data. Basically, the scheme stays as the above but the server sends 307, or other redirect code, with Accept-CH, so after the redirect the client adds the missing CH. The discussed drawback was that there could be an infinite loop if it’s implemented wrongly. Benefits of this approach is that the server doesn’t waste time on full page render, the response size is lower and that the server is sure that the browser will make the second request. Tried to optimize the first approach, with Critical-CH, to save as much as possible on the first request, the response that was 200 with empty page (don’t do it on prod!). But it is not future proof. Where browser may not send second request at all. Also guessing that browser supports UA-CH is *not defined* – just heuristic that “Sec-CH-UA-Mobile” is set, and the second guess is that one of the high entropy is missing is by maintaining on server side list of all HE-CH and comparing it to the request. This list is also constantly changing so this is also a *not well-defined* part of the specification. Also tested the Critical-CH together with 307. So, even if the guess that browser doesn’t support CH is wrong, the browser will still follow the redirect and render the required page to the user. For browsers that support CH the expected behavior was to just make a second request and ignore the response. And here was a surprise, there was no second request to the page, the browser just followed the redirect. This behavior was mentioned only in the source code thread, and not in the specs: https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/client_hints/README.md <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%2F%2B%2Frefs%2Fheads%2Fmain%2Fcomponents%2Fclient_hints%2FREADME.md&data=04%7C01%7Ckirillk%40top4.com%7C863c28c144d7423ac71c08da1e0a18a3%7C5add8cc5d607420c9cf4fc411e4cafd0%7C0%7C1%7C637855325982683054%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=CyoyqpC0hFT51CldbM30WGosZRU8aZ3cb%2BhEBPNLN08%3D&reserved=0> *“The Critical-CH retry mechanism is implemented as content::CriticalClientHintsThrottle and all of the important logic is in WillProcessResponse. When a retry situation is found (and the redirected_ flag isn’t set) the header is stored, the new hints are stored as normal and the request is “restarted” (i.e. the response is changed to an internal redirect to the same location, which is also what DevTools sees).”* This message is already got long, and there’re more of non-answered questions about CH that deserve a separate post. But my main point is that there’s no defined way how to get the required high entropy client hints without harming the user and the web site owner. Regards On Friday, March 25, 2022 at 5:23:03 PM UTC+3 James Rosewell wrote: > Adding API Owners. > > I appreciate that you will be subject to internal controls and that it may > be taking some time for you to go through the required training and > compliance. Could you let this group know when that is completed please? > > We are now in a different world where blink-dev are part of the public > reporting and monitoring agreement entered into between Google and the CMA. > We are keen to support your compliance efforts and are encouraging others > to understand the obligations you are now under. > > In relation to this specific proposal I’m concerned you made no mention of > the agreement or its status in relation to the commitments when you presented > it last week at the W3C <https://github.com/antifraudcg/meetings/issues/3> > and show every sign of progressing as if nothing has changed. > On Thursday, 24 March 2022 at 15:51:44 UTC mike...@chromium.org wrote: > >> Hi James, >> >> We are committed to achieving the purpose of the Privacy Sandbox >> commitments accepted by the CMA >> <https://blog.google/around-the-globe/google-europe/path-forward-privacy-sandbox/> >> >> in February 2022. We encourage feedback on how to better achieve that >> purpose through our technical proposals, and we will report publicly on >> feedback we receive as set out in the commitments. We are in constant >> dialogue with the CMA on these issues, and members of the web ecosystem are >> also welcome to discuss these issues with the CMA. We therefore hope that >> everyone will understand when we decline to participate in public >> discussions on legal or internal aspects of compliance with the >> commitments, or to detail our direct exchanges with the CMA. >> >> thanks, >> Mike >> >> On 3/17/22 10:53 AM, 'James Rosewell' via blink-dev wrote: >> >> Google and CMA reached an agreement on 4th February 2022 concerning >> Privacy Sandbox and Topics. The agreement requires Google to train staff >> in Annex 3 >> <https://assets.publishing.service.gov.uk/media/62052c6a8fa8f510a204374a/100222_Appendix_1A_Google_s_final_commitments.pdf>, >> >> and paragraph 4.119 >> <https://assets.publishing.service.gov.uk/media/62052c52e90e077f7881c975/Google_Sandbox_.pdf> >> >> states, *“Google has committed to instruct its staff and agents not to >> make claims to other market players that contradict the commitments, and to >> provide training to its relevant staff and agents to ensure that they are >> aware of the requirements of the Final Commitments”*. >> >> Paragraph 21 of the agreement states *“During the standstill period, the >> CMA may notify Google that competition law concerns remain such that the >> Purpose of the Commitments will not be achieved. Google will work with the >> CMA without delay to seek to resolve concerns raised and address comments >> made by the CMA with a view to achieving the Purpose of the Commitments. >> Google will inform the CMA of how it has responded to those comments”*. >> >> The standstill period is defined in the agreement as is User-Agent >> reduction. >> >> Will Google representatives confirm they have approval from the CMA to >> progress this proposal? If not then please can they confirm that the >> proposal is now delayed until such approval is provided? >> On Tuesday, 14 January 2020 at 11:22:00 UTC yo...@yoav.ws wrote: >> >>> Contact emails >>> >>> yoav...@chromium.org, aaro...@chromium.org >>> >>> Summary >>> >>> We want to freeze and unify (but not remove) the User Agent string in >>> HTTP requests as well as in `navigator.userAgent` >>> >>> Motivation >>> >>> The User-Agent string is an abundant source of passive fingerprinting >>> information about our users. It contains many details about the user’s >>> browser and device as well as many lies ("Mozilla/5.0", anyone?) that were >>> or are needed for compatibility purposes, as servers grew reliant on bad >>> User Agent sniffing. >>> >>> On top of those privacy issues, User-Agent sniffing is an abundant >>> source of compatibility issues, in particular for minority browsers, >>> resulting in browsers lying about themselves (generally >>> <https://www.zdnet.com/article/vivaldi-to-change-user-agent-string-to-chrome-due-to-unfair-blocking/> >>> >>> or to specific sites >>> <https://github.com/mozilla/webcompat-addon/blob/master/src/injections/js/bug1472075-bankofamerica.com-ua-change.js>), >>> >>> and sites (including Google properties) being broken >>> <https://www.onmsft.com/news/google-has-apparently-blocked-its-stadia-cloud-gaming-service-on-the-chromium-based-microsoft-edge> >>> >>> in some browsers for no good reason. >>> >>> The above abuse makes it desirable to freeze the UA string and replace >>> it with a better mechanism. There have been past attempts at UA string >>> freezing from the Safari team, but without an alternative way to perform UA >>> based content-negotiation, they had to be partially reverted. >>> >>> >>> >>> The User Agent Client Hints <https://wicg.github.io/ua-client-hints/> >>> (UA-CH) feature provides an alternate source for the information the >>> User-Agent string provides, both in its request header form as well as its >>> JS API one. >>> >>> Its main advantages are: >>> >>> - >>> >>> It provides the required information only when the server requests >>> it, over secure connections, making any fingerprinting that relies on it >>> be active >>> fingerprinting, which enables such use to be audited, as well as >>> acted-upon by the browser (e.g. in a future implementation of the >>> Privacy >>> Budget <https://github.com/bslassey/privacy-budget>). >>> - >>> >>> It provides the information in small increments, so servers are only >>> exposed to the information they need and request, rather than being >>> exposed >>> to the full gamut of the UA string even if they are just trying to >>> figure >>> out one detail about the browser. (e.g. brand and major version) >>> - >>> >>> Since it provides the information via dedicated fields, it enables >>> better ergonomics and makes it less likely for servers to get it wrong >>> and >>> cause compatibility issues. >>> - >>> >>> And finally, starting fresh will enable us to drop a lot of the >>> legacy baggage that the UA string carries (“Mozilla/5.0”, “like Gecko”, >>> “like KHTML”, etc) going forward. >>> >>> >>> Once UA-CH ships >>> <https://groups.google.com/a/chromium.org/d/msg/blink-dev/A4wxFpvqUfA/g7iccl9ICgAJ> >>> >>> as an alternative means for browser-specific content adaptation, we would >>> like to freeze the User-Agent string. >>> >>> We propose to deprecate at M81 (starting to emit console warnings in >>> pages that read that string in JS), freeze its version information at M83, >>> and unify strings of different devices at M85. See detailed freezing plan >>> below. >>> >>> This timeline provides 3 months for developers to move to the new >>> mechanism for their future browser and OS version needs, and 6 months for >>> more sophisticated OS or device specific targeting. >>> >>> Freezing plan >>> >>> Different parts of the UA string have different compatibility >>> implications. >>> >>> Some parts of it, such as the browser version and the OS version, can be >>> frozen without any backwards compatibility implications. Values that worked >>> in the past will continue to work in the future. >>> >>> Other parts, such as the model (for mobile devices) and the OS platform, >>> can have implications on sites that tailor their UI to the underlying OS or >>> that target a very specific model in their layout. Such sites will need to >>> migrate to use UA-CH. >>> >>> As such we are planning to freeze the parts that are amenable to >>> freezing fairly early, and gradually unify the rest. >>> >>> >>> Milestone >>> >>> Stable date >>> >>> Action >>> >>> M81 >>> >>> Mid March ‘20 >>> >>> Deprecate access to `navigator.userAgent` >>> >>> M83 >>> >>> Early June ‘20 >>> >>> Freeze browser version and unify OS versions >>> >>> M85 >>> >>> Mid September ‘20 >>> >>> Unify desktop OS string as a common value for desktop browsers. >>> >>> Unify mobile OS/device strings as a similarly common value for those at >>> M85 (*) >>> >>> >>> (*) For the mobile value, we may split it further based on common device >>> dimensions, as a one-time exercise, to reduce the compatibility risk of >>> unification. >>> >>> >>> >>> Interoperability and Compatibility Risk >>> >>> The compatibility risk varies at different stages. >>> >>> For the freezing planned for M83, the compatibility risk is low. >>> Existing UA sniffing code will continue to work as expected. It is only >>> future UA sniffing code that will need to change and use the UA client >>> hints instead. >>> >>> For the unification planned for M85, the compatibility risk is medium. >>> Some sites can modify their responses based on the OS and device model, and >>> those sites will have to change their UA sniffing code to use UA-CH. We >>> expect such sites to be well maintained (otherwise, how can they keep up >>> with OS UI and device model changes?). Therefore, having 4 releases to >>> modify their code seems sufficient. >>> >>> In the long term, we expect this change to improve compatibility, as UA >>> sniffing based on UA-CH is bound to be more reliable than the current >>> status quo. >>> >>> As for interoperability, we have other vendors on board with UA >>> freezing, but not necessarily with the UA Client Hints mechanism, that is >>> supposed to replace it. That can create a tricky situation, where >>> developers would need to rely on the User-Agent string for some browsers >>> and on UA-CH for others. >>> >>> Edge: Public support >>> <https://twitter.com/_scottlow/status/1206831008261132289> >>> >>> Firefox: Public support >>> <https://github.com/mozilla/standards-positions/issues/202#issuecomment-558294095> >>> >>> for freezing the UA string - “freezing the User Agent string without any >>> client hints—seems worth-prototyping” >>> >>> Safari: Shipped to some extent. Safari has attempted to completely >>> freeze <https://twitter.com/rmondello/status/943545865204989953> the UA >>> string in the past, without providing an alternative mechanism. That got a >>> lot of pushback, which resulted in somewhat reverting that decision. >>> Nowadays, their UA string seems frozen, other than updates to the OS >>> version and the browser major version. >>> >>> >>> Alternative implementation suggestion for web developers >>> >>> For many (most?) uses of UA sniffing today, a better tool for the job >>> would be to use feature detection. Where feature detection fails >>> developers, UA Client Hints <https://wicg.github.io/ua-client-hints/> >>> are the right path forward. >>> >>> Potential deployment hurdles compared to status quo: >>> >>> - >>> >>> Third party services that rely on the UA string would need to >>> convince the sites that include them to delegate that information to >>> them >>> using Feature Policy. >>> - >>> >>> The opt-in based mechanism of Client Hints currently suffers from >>> the fact that on the very-first view, the browser have not yet received >>> the >>> opt-in. That means that requiring specific UA information (e.g. device >>> model, platform) on the very-first navigation request may incur a delay. >>> We >>> are exploring options to enable the opt-in to happen further down the >>> stack >>> to avoid that delay. >>> >>> >>> Usage information from UseCounter >>> <https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/core/page/UseCounter.h&sq=package:chromium&type=cs&q=file:UseCounter.h%20Feature&l=39> >>> >>> If we were going to remove the User-Agent string outright, the answer to >>> that would’ve been “a lot!!!!111”. >>> >>> Given that we are planning to freeze it, we expect the “removal” to be >>> mostly backwards compatible, barring cases of specific OS or device >>> adaptation. >>> >>> Client-side UA sniffing use counters >>> <https://www.chromestatus.com/metrics/feature/timeline/popularity/2663> >>> show it’s being used by ~90% of sites. But again, since we’re talking about >>> freezing, it should not break most uses. >>> >>> Entry on the feature dashboard <https://www.chromestatus.com/> >>> >>> https://www.chromestatus.com/feature/5704553745874944 >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to blink-dev+...@chromium.org. >> >> To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6e72d610-9978-4d7a-862a-5830edbf6b81n%40chromium.org >> >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6e72d610-9978-4d7a-862a-5830edbf6b81n%40chromium.org?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/575ebed2-697a-4169-84d4-5dce9a433fd7n%40chromium.org.
