Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
Hi Colin, > On Apr 26, 2018, at 8:57 AM, Colin Bendell | +1.613.914.3387 >wrote: > > […] images to a device. I have a laundry list of other examples where the > server might actually know more than the client to prevent a) a broken > user experience and b) prevent cellular data waste. Would you mind documenting those items in a Bugzilla report? We are interested in specific use cases that suffer with a frozen user agent string, and what (apparently bad) techniques you need to use to work around it. Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
I can totally understand your intentions, it feels like that in 2018 UA should not be a relevant piece of information, and especially not be a factor in the decision if a server should serve a website to a certain request. But UA is how the Internet currently works. Correct me if I'm wrong, but it feels like you are trying to change the Internet. Are you trying to get rid of it for good? I understand that these kind of drastic, revolutionary, changes might take casualties, I feel like the industry was not given a chance to adopt to this change. Sent: Thursday, April 26, 2018 at 6:22 PM From: "Michael Catanzaro" <mcatanz...@igalia.com> To: "Ricky Young" <ri...@gmx.com> Cc: "Anne van Kesteren" <ann...@annevk.nl>, "webkit-dev@lists.webkit.org" <webkit-dev@lists.webkit.org> Subject: Re: [webkit-dev] New iOS versions sending bogus User-Agent build data On Thu, Apr 26, 2018 at 7:15 AM, Ricky Young <ri...@gmx.com> wrote: > I guess that soon it won't be possible to determine iOS version > distribution using User-Agent sniffing. Right? Is there an > alternative for the browser to tell the iOS version? Hi, The very point of no longer sending the verison in the user agent is to make it difficult for websites to determine the operating system version. We already have to deal with enough serious web compatibility problems caused by the rest of the information in the user agent. I'm pleased that the version number is now stable -- that's a huge improvement -- but I won't be happy until Safari and other major browsers start sending fully-randomized UAs, or else a stub or blank UA ("Mozilla/5.0"). Michael ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On 26 April 2018 at 14:25, Ali Jumawrote: > It's worth noting that https://bugs.webkit.org/show_bug.cgi?id=182629 added > back the OS version to the UA string, at least on trunk (the reasons given > there, in https://bugs.webkit.org/show_bug.cgi?id=182629#c6, sound exactly > like those mentioned by Colin in this thread). Thanks for the catch. I had missed this commit. This is helpful. On 26 April 2018 at 15:40, Michael Catanzaro wrote: > I would say it's the most serious web compatibility problem that exists > today. Our users complain and blame us when important websites are broken in > WebKit because of it. I have personally wasted days [1] of [2] development > [3] effort [4] playing with WebKit's user agent quirks to get important > websites to work properly. Yea, I get that. The current state of the UA is terrible. It's really an archeological map of browser evolution. It's like going on a first-date and the first thing you talk about is your family tree, and how great-grandpa-andreesen who is all but estranged from the family. It is a tad embarrassing :) On 26 April 2018 at 15:40, Michael Catanzaro wrote: > On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell wrote: >> Again I ask, is there room for compromise where we can expose the >> version details in the UA (or some alternative) so that we ensure a >> consistent and optimized user experience? > > I don't know. I wish there was, but I don't think so. If you have any > suggestions, that'd be great, but I think it's going to be extremely > difficult or impossible to solve this problem in a way that makes everyone > happy. I was only partially joking about the need for a UA2 spec. Perhaps it is time to bring this to one of the w3c working groups. I'll take it off this thread though :) /colin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell | +1.613.914.3387wrote: Can you give me an example where UA parsing is punishing users of alternative user agents? Is this a theoretical problem, or a widespread problem? I'm not asking to be divisive, but because I know for a fact that UA parsing is improving the user experience. I can give you dozens of examples where we must resort to UA parsing for the betterment of the user (for all flavors of UAs). I would say it's the most serious web compatibility problem that exists today. Our users complain and blame us when important websites are broken in WebKit because of it. I have personally wasted days [1] of [2] development [3] effort [4] playing with WebKit's user agent quirks to get important websites to work properly. You can look at the history of our quirks list for non-Safari ports [5] to get an idea of the trouble we've had to deal with in recent years. That doesn't count cases like [6] where I suspect user agent trouble, but just don't care to investigate. As a web engine developer, it's hard to understate how frustrating this is, especially in cases like [7] where we had tons of users complaining and the quirks were particularly difficult to write. It sometimes feels like website developers are our enemy, just out to break our web engine. Sometimes that's even true, e.g. when websites intentionally decide to block access to unrecognized browsers, or scare our users with claims that a browser is "unsupported" even though it works fine with a UA quirk. This is not a healthy situation for the web. If everyone was careful and responsible with how they use the user agent, it wouldn't be a problem, but at this point it's long been spoiled for everyone. I'm sorry that this fallout affects developers like you who are just trying to work around a bug. :( On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell | +1.613.914.3387 wrote: Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience? I don't know. I wish there was, but I don't think so. If you have any suggestions, that'd be great, but I think it's going to be extremely difficult or impossible to solve this problem in a way that makes everyone happy. I don't think I'll be happy until major browsers send the same, standardized user agent as other major browsers. (Or send fully-randomized user agents, but that's probably an impossible dream.) Freezing just the version numbers is not good enough, but it's a step in the right direction, and I really appreciate it. [1] https://trac.webkit.org/changeset/206519/webkit [2] https://trac.webkit.org/changeset/216139/webkit [3] https://trac.webkit.org/changeset/216343/webkit [4] https://trac.webkit.org/changeset/217203/webkit [5] https://trac.webkit.org/log/webkit/trunk/Source/WebCore/platform/UserAgentQuirks.cpp [6] https://bugs.webkit.org/show_bug.cgi?id=180995 [7] https://bugs.webkit.org/show_bug.cgi?id=171770#c4 ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
It's worth noting that https://bugs.webkit.org/show_bug.cgi?id=182629 added back the OS version to the UA string, at least on trunk (the reasons given there, in https://bugs.webkit.org/show_bug.cgi?id=182629#c6, sound exactly like those mentioned by Colin in this thread). On Thu, Apr 26, 2018 at 1:49 PM Colin Bendell | +1.613.914.3387 < co...@bendell.ca> wrote: > On 26 April 2018 at 13:18, Konstantin Tokarevwrote: > > 26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387 <(613)%20914-3387>" > : > >> On 26 April 2018 at 12:23, Konstantin Tokarev > wrote: > >>> Not to mention those evil people who reject page loading for user > agents > >>> they don't (want to) support > >> > >> How does locking the UA solve the misbehaving parsers in the wild? > > > > There will be less information in UA and therefore less reasons for > developers > > to do this, instead of using direct feature testing > > > >> They will still misbehave and break the user experience. However, > >> those of us that are trying to optimize the user experience by working > >> around bugs for specific versions are now handicapped and punished. > > > > While parsing UA string handicaps and punishes users of alternative user > agents. > > Can you give me an example where UA parsing is punishing users of > alternative user agents? Is this a theoretical problem, or a > widespread problem? I'm not asking to be divisive, but because I know > for a fact that UA parsing is improving the user experience. I can > give you dozens of examples where we must resort to UA parsing for the > betterment of the user (for all flavors of UAs). > > Do you have any alternatives other than "more javascript"? > > Again I ask, is there room for compromise where we can expose the > version details in the UA (or some alternative) so that we ensure a > consistent and optimized user experience? > > /colin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On 26 April 2018 at 13:18, Konstantin Tokarevwrote: > 26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387" : >> On 26 April 2018 at 12:23, Konstantin Tokarev wrote: >>> Not to mention those evil people who reject page loading for user agents >>> they don't (want to) support >> >> How does locking the UA solve the misbehaving parsers in the wild? > > There will be less information in UA and therefore less reasons for developers > to do this, instead of using direct feature testing > >> They will still misbehave and break the user experience. However, >> those of us that are trying to optimize the user experience by working >> around bugs for specific versions are now handicapped and punished. > > While parsing UA string handicaps and punishes users of alternative user > agents. Can you give me an example where UA parsing is punishing users of alternative user agents? Is this a theoretical problem, or a widespread problem? I'm not asking to be divisive, but because I know for a fact that UA parsing is improving the user experience. I can give you dozens of examples where we must resort to UA parsing for the betterment of the user (for all flavors of UAs). Do you have any alternatives other than "more javascript"? Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience? /colin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387": > On 26 April 2018 at 12:23, Konstantin Tokarev wrote: >> Not to mention those evil people who reject page loading for user agents >> they don't (want to) support > > How does locking the UA solve the misbehaving parsers in the wild? There will be less information in UA and therefore less reasons for developers to do this, instead of using direct feature testing > They will still misbehave and break the user experience. However, > those of us that are trying to optimize the user experience by working > around bugs for specific versions are now handicapped and punished. While parsing UA string handicaps and punishes users of alternative user agents. -- Regards, Konstantin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On 26 April 2018 at 12:23, Konstantin Tokarevwrote: > 26.04.2018, 19:16, "Michael Catanzaro" : >> On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzaro >> wrote: >>> By fixing the WebKit bug, of course. And in the meantime you can work >>> around it on the server side by not using img src=mp4, right? Today, we can work around it by ignoring the Accept field. However, tomorrow once the bug is fixed we will be in the paradox of not knowing which versions of Safari are telling the truth. >> Consider the other perspective on this problem. If other servers look >> at the WebKit version in the UA to determine if WebKit supports img >> src=mp4, other WebKit ports that don't support this are going to be out >> of luck and get broken pages. I know that's not what you're doing -- >> you're looking at iOS version instead, and only doing it to work around >> a specific bug, which is much better -- but the problem of websites >> sending bad content based on bad user agent parsing is so severe that >> we don't have many good options, here. :/ > > Not to mention those evil people who reject page loading for user agents > they don't (want to) support How does locking the UA solve the misbehaving parsers in the wild? They will still misbehave and break the user experience. However, those of us that are trying to optimize the user experience by working around bugs for specific versions are now handicapped and punished. The reality is that bugs exist. There is a give and take here where the content negotiation is sometimes smarter on the client, and sometimes smarter on the server. In the case of media, the server is usually smarter and must be because of the long tail of adoption. Ideally everyone on the planet will adopt to ios V.Latest overnight, but sadly this will not be the case. Servers must account for the long tail of users on many different versions. Right now we are forced to decide to apply a solution that works only for v.latest, or apply a solution that works for v.lowest-common-denominator. In both scenarios we compromise the user experience. Ideally we wouldn't have to sacrifice some users and would be able to account for bugs in the wild. As I said before, if this detail is available in javascript, then why hide it on the wire? Perhaps it's time for a User-Agent2 with precise feature declarations? Even still a version number is needed because, well, bugs. While I understand the fingerprinting concern, I'm not particularly convinced that UA obfuscation is the best solution for users and the user experience. Without opening up the religious debate of fingerprinting with the UA, how can provide enough detail (version numbers) so that service providers can work toward the greater good of improving the user experience? Is there room for compromise? /colin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
26.04.2018, 19:16, "Michael Catanzaro": > On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzaro > wrote: >> By fixing the WebKit bug, of course. And in the meantime you can work >> around it on the server side by not using img src=mp4, right? > > Consider the other perspective on this problem. If other servers look > at the WebKit version in the UA to determine if WebKit supports img > src=mp4, other WebKit ports that don't support this are going to be out > of luck and get broken pages. I know that's not what you're doing -- > you're looking at iOS version instead, and only doing it to work around > a specific bug, which is much better -- but the problem of websites > sending bad content based on bad user agent parsing is so severe that > we don't have many good options, here. :/ Not to mention those evil people who reject page loading for user agents they don't (want to) support -- Regards, Konstantin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzarowrote: By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using img src=mp4, right? Consider the other perspective on this problem. If other servers look at the WebKit version in the UA to determine if WebKit supports img src=mp4, other WebKit ports that don't support this are going to be out of luck and get broken pages. I know that's not what you're doing -- you're looking at iOS version instead, and only doing it to work around a specific bug, which is much better -- but the problem of websites sending bad content based on bad user agent parsing is so severe that we don't have many good options, here. :/ Michael ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On Thu, Apr 26, 2018 at 10:57 AM, Colin Bendell | +1.613.914.3387wrote: How do we navigate this? By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using , right? Michael ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
I'm running into this issue right now with the locked UA. Here are the challenges: 1. ios 11.3 safari bug broke despite advertising `accept: video/*` in the request. (see bug 185029). I now need to use UA detection to patch versions of a response based on existing bugs. In short, we as an industry, depend on UA parsing to work around bugs. This happened last year when the jp2k support increased to 10bit depth and the decoder was inflating the 8bit colour pallet to 10bit. We needed to use UA parsing to determine if it was safe to send jp2k images to a device. I have a laundry list of other examples where the server might actually know more than the client to prevent a) a broken user experience and b) prevent cellular data waste. 2. Javascript based feature detection is inefficient and slows down safari' preloader. The preloader is one of the best performance enhancements to come to the modern browser. Using javscript for feature detection is a step backward for the preloader. Webkit does not provide enough declarations of its feature set in HTTP headers so we resort to UA device characteristics. If the features are available with javascript inquiry, then how is that any different than exposing these features at request time? In short, we are stuck. blocking the UA like this will degrading the user experience because it will force us to either not take advantage of cell-data/cpu/battery improvements (like sending mp4s instead of animated gifs) or it will force us to use javascript and likewise degrade the user experience with slower web experiences. How do we navigate this? /colin On 26 April 2018 at 11:22, Michael Catanzarowrote: > On Thu, Apr 26, 2018 at 7:15 AM, Ricky Young wrote: >> >> I guess that soon it won't be possible to determine iOS version >> distribution using User-Agent sniffing. Right? Is there an alternative for >> the browser to tell the iOS version? > > > Hi, > > The very point of no longer sending the verison in the user agent is to make > it difficult for websites to determine the operating system version. > > We already have to deal with enough serious web compatibility problems > caused by the rest of the information in the user agent. I'm pleased that > the version number is now stable -- that's a huge improvement -- but I won't > be happy until Safari and other major browsers start sending > fully-randomized UAs, or else a stub or blank UA ("Mozilla/5.0"). > > Michael > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
Actually it does, a lot! I guess that soon it won't be possible to determine iOS version distribution using User-Agent sniffing. Right? Is there an alternative for the browser to tell the iOS version? Sent: Thursday, April 26, 2018 at 1:54 PM From: "Anne van Kesteren" <ann...@annevk.nl> To: "Ricky Young" <ri...@gmx.com> Cc: "Konstantin Tokarev" <annu...@yandex.ru>, "webkit-dev@lists.webkit.org" <webkit-dev@lists.webkit.org> Subject: Re: [webkit-dev] New iOS versions sending bogus User-Agent build data On Thu, Apr 26, 2018 at 12:30 PM, Ricky Young <ri...@gmx.com> wrote: > I read the report and still find it hard to understand, if "User Agent > sniffing is a terrible way to determine whether a browser supports certain > features", what is the correct way of doing it? https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Feature_detection Hope that helps, -- https://annevankesteren.nl/[https://annevankesteren.nl/] ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On Thu, Apr 26, 2018 at 12:30 PM, Ricky Youngwrote: > I read the report and still find it hard to understand, if "User Agent > sniffing is a terrible way to determine whether a browser supports certain > features", what is the correct way of doing it? https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Feature_detection Hope that helps, -- https://annevankesteren.nl/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
Hi Konstantin, Thanks for your prompt response :-) I didn't notice this bug and the respective patches. I read the report and still find it hard to understand, if "User Agent sniffing is a terrible way to determine whether a browser supports certain features", what is the correct way of doing it? Sent: Thursday, April 26, 2018 at 1:16 PM From: "Konstantin Tokarev" <annu...@yandex.ru> To: "Ricky Young" <ri...@gmx.com>, "webkit-dev@lists.webkit.org" <webkit-dev@lists.webkit.org> Subject: Re: [webkit-dev] New iOS versions sending bogus User-Agent build data 26.04.2018, 13:08, "Ricky Young" <ri...@gmx.com>: > Until iOS 11.3, User-Agent build data contained the build that corresponds to > the device (e.g. Mobile/15D100 if the requesting device was running iOS > 11.2.6). > > Since iOS 11.3 was released I noticed that the User-Agent that is reported to > the brower is reporting a bogus build, i.e Mobile/15E148, regardless to the > iOS version that the requesting device is running. I would expect that a > device that runs 11.3 will reprot a different User-Agent than a device that > runs 11.3.1 and certianly than a device that runs 11.4 (Beta 1 & 2). > > I just wonder if my observation is an expected behavior, and if so, what is > the intention behind it. If this behavior is not intended and expected, I > would love to research it and propose a patch (would be my first Open Source > contribution). Hi, User-Agent string is frozen since https://bugs.webkit.org/show_bug.cgi?id=180365 -- Regards, Konstantin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
26.04.2018, 13:08, "Ricky Young": > Until iOS 11.3, User-Agent build data contained the build that corresponds to > the device (e.g. Mobile/15D100 if the requesting device was running iOS > 11.2.6). > > Since iOS 11.3 was released I noticed that the User-Agent that is reported to > the brower is reporting a bogus build, i.e Mobile/15E148, regardless to the > iOS version that the requesting device is running. I would expect that a > device that runs 11.3 will reprot a different User-Agent than a device that > runs 11.3.1 and certianly than a device that runs 11.4 (Beta 1 & 2). > > I just wonder if my observation is an expected behavior, and if so, what is > the intention behind it. If this behavior is not intended and expected, I > would love to research it and propose a patch (would be my first Open Source > contribution). Hi, User-Agent string is frozen since https://bugs.webkit.org/show_bug.cgi?id=180365 -- Regards, Konstantin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev