Re: Fixing QNetworkAccessManager use
On Thu, Feb 20, 2020 at 9:58 AM Friedrich W. H. Kossebau wrote: > > Am Mittwoch, 19. Februar 2020, 21:01:20 CET schrieb Johan Ouwerkerk: > > On Wed, Feb 19, 2020 at 2:09 PM Friedrich W. H. Kossebau > > > > wrote: > > > Personally I am still unsure what the actual issue is. Why are redirects > > > needed at all. Why all the address changes all the time? > > > > It is part of the HTTP spec for servers to be able to inform clients > > that resource /foo/bar has moved to /bar/baz, either temporarily or > > permanently. > > :) Thanks for that explanation, but that was not my question here (that part I > am well aware of, done my share of web stuff). > > It was rather: why are subdomain names and/or access paths not once properly > designed, but instead changed so often that redirection seems so important to > be a default feature? Just because one can? Things don't change extremely often. Sometimes however requirements or other factors change, which necessitates changing where a resource is hosted. When this happens, it is extremely useful to have the ability to relocate it elsewhere. To use an example, when we first setup files.kde.org it was used by a couple of things, including Necessitas for the Qt binaries that get downloaded on to end user (Android) devices. When this was first established, traffic was well within the reasonable bounds we had expected when setting this up, and everything was served directly by our (single) server. This went quite well for a while. Sometime a bit later, an application was released on Google Play that used Necessitas which was *extremely* popular, to the extent it caused around a terabyte of data to be used within 48 hours or so. Hetzner bandwidth was at this time not only limited to 100mbps, but also capped - with the limit being 5 TB per month and overage after that resulting in a charge per terabyte. We therefore made the decision to convert files.kde.org to a mirror network (like was already in place for download.kde.org), with redirection taking place using Mirrorbrain. We were able to complete this transition quickly thanks to the generous support of some of our mirrors who established mirrors of files.kde.org. Fortunately Necessitas had full support for handling redirects, so this is something we were able to accomplish without any issues. Had redirect support not been available, we would have been left with no way out at that time. I also have other examples involving Marble (including where we got bitten by QNetworkAccessManager for the very first time - back in January 2012) and numerous other KDE Edu applications (all of which fortunately avoided QNAM). > When we write code, we try to keep API stable as much as possible, and only > change API when really useful, and that means for the consumer. When doing > references in text we try to have eternally stable pointers (thanks ISBN & > Co.), > > But this request for stable URLs on the internet might be an idealistic fight > against windmills of a web 1.0 person... > > Cheers > Friedrich > > Cheers, Ben
Re: Review Kid3
El dimecres, 19 de febrer de 2020, a les 6:39:50 CET, Urs Fleisch va escriure: > Hi Albert, > > > You don't need all that > > echo "Extracting po/en/kid3_qt.po" > > part, scripty just does it every single day for you > > I saw that an English po is now generated and downloaded when using > translations/download-pos.sh. In this case the file po/en/kid3_qt.po > already exists and the "Extracting po/en/kid3_qt.po" part is not executed, > so it does not harm. What could be improved in scripty's job is the fact > that it generates a full English po with a lot of wrong "translations" > therein (fuzzy, guessed incorrectly by msgmerge). msgmerge should not be > called on po files which only contain plurals. But anyway, I explicitly > call lupdate with the -pluralonly option in translations/po2ts.py, so the > wrong translations do not harm in the end. > > > I don't understand why you have that po2ts.py either, we have an > extra-cmake-modules > > thing that will make it for you (ECMCreateQmFromPoFiles.cmake), you > should be using it :) > > I tried it with ECMPoQmTools, but unfortunately, it didn't work in this > case. I documented this in the header of translations/CMakeLists.txt: > > # Using ecm_install_po_files_as_qm() from ECMPoQmTools does not work > # because the generated ts files do not have context information, > # just one single context with an empty name. So what you are saying is that ecm_install_po_files_as_qm doesn't work at all? That's strange since it works for lots of KDE Frameworks that use it. In the future, please when you find what seems to be a bug like that tell us about it so we can make it work and you don't have to do your own thing. Cheers, Albert > > Regards, > Urs >
Re: Banning QNetworkAccessManager
El dimecres, 19 de febrer de 2020, a les 9:28:22 CET, Volker Krause va escriure: > Additionally, improved documentation, a possible KNAM and/or driving the QNAM > changes upstream can still be done alongside this obviously. Also for Qt5 which will probably never be changed a clazy warning and making it easy to run clazy on gitlab CI would be great. Cheers, Albert > > Regards, > Volker
Re: Banning QNetworkAccessManager
El dimecres, 19 de febrer de 2020, a les 8:05:01 CET, Ben Cooksley va escriure: > The easiest path forward is to simply ban QNetworkAccessManager. Stop saying that, that's not going to happen. Cheers, Albert > > > > > Regards, > > Volker > > Regards, > Ben >
Re: Fixing QNetworkAccessManager use
Am Mittwoch, 19. Februar 2020, 21:01:20 CET schrieb Johan Ouwerkerk: > On Wed, Feb 19, 2020 at 2:09 PM Friedrich W. H. Kossebau > > wrote: > > Personally I am still unsure what the actual issue is. Why are redirects > > needed at all. Why all the address changes all the time? > > It is part of the HTTP spec for servers to be able to inform clients > that resource /foo/bar has moved to /bar/baz, either temporarily or > permanently. :) Thanks for that explanation, but that was not my question here (that part I am well aware of, done my share of web stuff). It was rather: why are subdomain names and/or access paths not once properly designed, but instead changed so often that redirection seems so important to be a default feature? Just because one can? When we write code, we try to keep API stable as much as possible, and only change API when really useful, and that means for the consumer. When doing references in text we try to have eternally stable pointers (thanks ISBN & Co.), But this request for stable URLs on the internet might be an idealistic fight against windmills of a web 1.0 person... Cheers Friedrich
Re: Fixing QNetworkAccessManager use
On Wed, Feb 19, 2020 at 2:09 PM Friedrich W. H. Kossebau wrote: > > Personally I am still unsure what the actual issue is. Why are redirects > needed at all. Why all the address changes all the time? > It is part of the HTTP spec for servers to be able to inform clients that resource /foo/bar has moved to /bar/baz, either temporarily or permanently. This can be used to do things like mapping /retrieve/document/by/alias -> /documents/actual/document-id, or to redirect to different hosts entirely, or to inform plain text HTTP clients to upgrade to using HTTPS instead. (HSTS is a spec describing how a server can then ask the client to subsequently enforce its policy preference for when to connect over HTTPS.) The main difference between temporary and permanent redirects is that clients are allowed to "remember" when a resource moved in the case of permanent redirects so they can optimise subsequent calls to the moved resources (bypassing the redirect entirely). But as you can see, the temporary redirect is something that could be used to do load balancing: assume /resource is expensive to compute or retrieve, then put a proxy in front which load balances to the actual pool of servers using temporary redirects. (Of course you could argue that in such a case maybe round-robin DNS is a better solution altogether.) Regards, - Johan
Re: Fixing QNetworkAccessManager use
Am Mittwoch, 19. Februar 2020, 08:05:01 CET schrieb Ben Cooksley: > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > It would also help to know where specifically we have that problem, so we > > can actually solve it, and so we can figure out why we failed to fix this > > there earlier. > > Just bringing this up again - it seems we've not had much movement on > this aside from the Wiki page. The wiki page currently still just recommends to set "networkAccessManger->setAttribute(QNetworkRequest::FollowRedirectsAttribute, true);" Which seems simple, but possible not what is enough in all cases. So my open questions here to be able to act on code I contribute to are: a) What about the mentioned QNetworkRequest::NoLessSafeRedirectPolicy, in which cases should that be used and when not? b) What about the HSTS stuff, when is that recommended? c) What is a sane number for QNetworkRequest::maximumRedirectsAllowed? Both in general and when it comes to KDE servers. Personally I am still unsure what the actual issue is. Why are redirects needed at all. Why all the address changes all the time? The "U" in "URL"/"URI" is for "uniform", not "unstable", isn't it ;) Can you give some examples for URLs of resources our code uses on KDE servers, and why they needed to change? And if those redirects are permanent, should the client side not also permanently update to the new location then, instead of continuing to poke the old address every time again and again, until one day it will poke into a void because the backward compat redirect support has been dropped? Cheers Friedrich
Re: Banning QNetworkAccessManager
On woensdag 19 februari 2020 13:38:55 CET Johnny Jazeix wrote: > Hi, > > as Volker said, without any alternative, we can't just ban QNAM. > And even with one, we would need time to implement it (for example, > for GCompris, this part of the code was written by someone who is less > active now, so rewriting it, testing it and be sure it works will take > some time). > > Can't we use lxr to find all applications using it, check if they use > it in a good way or not instead? Yes... We're also using it in Krita. It has been there for many years now, to get Krita's rss feed. -- https://www.krita.org
Re: Banning QNetworkAccessManager
Hi, as Volker said, without any alternative, we can't just ban QNAM. And even with one, we would need time to implement it (for example, for GCompris, this part of the code was written by someone who is less active now, so rewriting it, testing it and be sure it works will take some time). Can't we use lxr to find all applications using it, check if they use it in a good way or not instead? Johnny Le mer. 19 févr. 2020 à 10:05, Ben Cooksley a écrit : > > On Wed, Feb 19, 2020 at 9:30 PM Volker Krause wrote: > > > > On Wednesday, 19 February 2020 08:05:01 CET Ben Cooksley wrote: > > > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > > > I agree on the problem of QNAM's default, see also > > > > https://conf.kde.org/en/ > > > > akademy2019/public/events/135 on that subject. > > > > > > > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > > > > [...] > > > > > > > > > Prior to now, i've taken the approach of advertising that > > > > > QNetworkAccessManager is broken and needs a flag set within > > > > > applications when used, however unfortunately this seems to have been > > > > > an ineffective approach and cases of broken behaviour continue to > > > > > appear (despite this brokeness being known about and advertised since > > > > > back in the Qt 4.x series when this class was introduced) > > > > > > > > > > Therefore, given the Qt Project is unlikely to change their position > > > > > > > > > on this, I would like to propose the following: > > > > The Qt Project is actually in favor of changing at least the redirection > > > > default to exactly what we want it to be. > > > > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, > > > > and > > > > got approval both by Lars and one of the maintainers. It is merely stuck > > > > in dealing with the unit test fallout in Qt's CI that somebody has to > > > > take care of. Doesn't immediately help us of course, but claiming Qt is > > > > unwilling to change this is simply wrong. > > > > > > > > > 1) That effective immediately, QNetworkAccessManager and it's children > > > > > classes be banned and prohibited within KDE software, to be enforced > > > > > by server side hooks. > > > > > 2) That we fork QNetworkAccessManager and the associated classes > > > > > within the appropriate Framework (to be determined), where the > > > > > defective behaviour can then be corrected. > > > > > > > > While this might solve the redirection problem, it brings us yet another > > > > then unmaintained HTTP implementation next to the one in KIO, which is a > > > > far bigger problem long term. We need less of those, not more. > > > > > > > > To address the problem of people not using the correct defaults, I did > > > > propose a much less extreme approach in the above mentioned talk at > > > > Akademy, namely having a KNetworkAccessManager as a sub-class of QNAM in > > > > a low-tier frameworks that essentially just enables redirects and HSTS. > > > > QNAM's implementation isn't the problem after all, the defaults are. > > > > > > > > Commit hooks warning about QNAM usage might still be a good idea, but > > > > as a > > > > warning only. Hard blocking QNAM-using code would block at least three > > > > repos I spend a lot of time on currently, none of which even talks to > > > > KDE > > > > servers. > > > > > > > > If you need to take more drastic measures against code not doing this > > > > correctly regardless, you can do that my dropping the server-side > > > > workarounds. Yes, that would break the affected application possibly, > > > > but > > > > at least it would not cause massive collateral damage for the people > > > > using this correctly. > > > > > > > > It would also help to know where specifically we have that problem, so > > > > we > > > > can actually solve it, and so we can figure out why we failed to fix > > > > this > > > > there earlier. > > > > > > Just bringing this up again - it seems we've not had much movement on > > > this aside from the Wiki page. > > > > > > Examining that Qt merge request, it not only is targeted at just Qt > > > 6.x, it also hasn't had any movement since the start of October 2019 - > > > 4 months ago. > > > Given that we are going to be on Qt 5.x for some time, that merge > > > request can't be considered to be the 'fix' for this issue. > > > > The targeting of Qt6 is due to this aiming at dev when that was still Qt > > 5.x, > > but you are right of course, somebody needs to do the work there to drive > > this > > to completion, no matter in which version it will actually land. > > Indeed. > > > > > > We need a solution that will ensure software released today doesn't > > > cause us pain in several years time, because as I illustrated in an > > > earlier email to this thread, software has a habit of remaining in use > > > for an extremely long amount of time (August 2014 being the release > > > date of the Marble versions in question hitting that old URL). > > > > > > The easiest
Re: Banning QNetworkAccessManager
On Wed, Feb 19, 2020 at 9:30 PM Volker Krause wrote: > > On Wednesday, 19 February 2020 08:05:01 CET Ben Cooksley wrote: > > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > > I agree on the problem of QNAM's default, see also > > > https://conf.kde.org/en/ > > > akademy2019/public/events/135 on that subject. > > > > > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > > > [...] > > > > > > > Prior to now, i've taken the approach of advertising that > > > > QNetworkAccessManager is broken and needs a flag set within > > > > applications when used, however unfortunately this seems to have been > > > > an ineffective approach and cases of broken behaviour continue to > > > > appear (despite this brokeness being known about and advertised since > > > > back in the Qt 4.x series when this class was introduced) > > > > > > > > Therefore, given the Qt Project is unlikely to change their position > > > > > > > on this, I would like to propose the following: > > > The Qt Project is actually in favor of changing at least the redirection > > > default to exactly what we want it to be. > > > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, and > > > got approval both by Lars and one of the maintainers. It is merely stuck > > > in dealing with the unit test fallout in Qt's CI that somebody has to > > > take care of. Doesn't immediately help us of course, but claiming Qt is > > > unwilling to change this is simply wrong. > > > > > > > 1) That effective immediately, QNetworkAccessManager and it's children > > > > classes be banned and prohibited within KDE software, to be enforced > > > > by server side hooks. > > > > 2) That we fork QNetworkAccessManager and the associated classes > > > > within the appropriate Framework (to be determined), where the > > > > defective behaviour can then be corrected. > > > > > > While this might solve the redirection problem, it brings us yet another > > > then unmaintained HTTP implementation next to the one in KIO, which is a > > > far bigger problem long term. We need less of those, not more. > > > > > > To address the problem of people not using the correct defaults, I did > > > propose a much less extreme approach in the above mentioned talk at > > > Akademy, namely having a KNetworkAccessManager as a sub-class of QNAM in > > > a low-tier frameworks that essentially just enables redirects and HSTS. > > > QNAM's implementation isn't the problem after all, the defaults are. > > > > > > Commit hooks warning about QNAM usage might still be a good idea, but as a > > > warning only. Hard blocking QNAM-using code would block at least three > > > repos I spend a lot of time on currently, none of which even talks to KDE > > > servers. > > > > > > If you need to take more drastic measures against code not doing this > > > correctly regardless, you can do that my dropping the server-side > > > workarounds. Yes, that would break the affected application possibly, but > > > at least it would not cause massive collateral damage for the people > > > using this correctly. > > > > > > It would also help to know where specifically we have that problem, so we > > > can actually solve it, and so we can figure out why we failed to fix this > > > there earlier. > > > > Just bringing this up again - it seems we've not had much movement on > > this aside from the Wiki page. > > > > Examining that Qt merge request, it not only is targeted at just Qt > > 6.x, it also hasn't had any movement since the start of October 2019 - > > 4 months ago. > > Given that we are going to be on Qt 5.x for some time, that merge > > request can't be considered to be the 'fix' for this issue. > > The targeting of Qt6 is due to this aiming at dev when that was still Qt 5.x, > but you are right of course, somebody needs to do the work there to drive this > to completion, no matter in which version it will actually land. Indeed. > > > We need a solution that will ensure software released today doesn't > > cause us pain in several years time, because as I illustrated in an > > earlier email to this thread, software has a habit of remaining in use > > for an extremely long amount of time (August 2014 being the release > > date of the Marble versions in question hitting that old URL). > > > > The easiest path forward is to simply ban QNetworkAccessManager. > > That might be the easiest path for you, but certainly not for me, given two of > the modules I work on most atm are using QNAM (not even to talk to KDE > infrastructure), and would suddenly no longer be allowed to be hosted here? Ideally they would move to whatever replacement we have for QNAM should it be banned :) > > Also, I have yet to see a suggestion for a viable alternative to QNAM. The > initially proposed fork would mean trading the possibility of broken redirect > handling for an unmaintained HTTP stack, I don't think that's a good deal for > the security of our users. > It isn't perfect yes. Sadly however the nega
Re: Review Kid3
Urs Fleisch ha scritto: > Hi Christoph, > >>> You should change the bug reporting to use bugzilla not your email address > in KAboutData >> >> Right. Urs, could you please request creation of a bugzilla product? > > I discussed the migration of the existing bug reports, feature requests and > help form items with Jonathan, and we agreed that GitLab issues > (https://invent.kde.org/kde/kid3/issues) are used for this purpose. All > existing issues from SourceForge have been migrated to GitLab and the source > code is fixed to point to this location. This is not a decision that a single project could take alone. I believe some people (including Jonathan) have good reasons for moving to gitlab, but as a community we can't have a split triaging process. If we move, we should move every project, and at this point of the gitlab migration there are still other issues. Moreover, we should make sure of our tooling works with gitlab too in addition to bugzilla and handle the transition. It is a project which require works and it will take time. Also, issue handling in gitlab is more limited then what bugzilla can provide, and by the time we settle down with the gitlab migration, bugzilla may have also catched up with some of the missing features. The usage of bugzilla is not under discussion right now for any project. -- Luigi
Re: Banning QNetworkAccessManager
On Wednesday, 19 February 2020 08:05:01 CET Ben Cooksley wrote: > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > I agree on the problem of QNAM's default, see also > > https://conf.kde.org/en/ > > akademy2019/public/events/135 on that subject. > > > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > > [...] > > > > > Prior to now, i've taken the approach of advertising that > > > QNetworkAccessManager is broken and needs a flag set within > > > applications when used, however unfortunately this seems to have been > > > an ineffective approach and cases of broken behaviour continue to > > > appear (despite this brokeness being known about and advertised since > > > back in the Qt 4.x series when this class was introduced) > > > > > > Therefore, given the Qt Project is unlikely to change their position > > > > > on this, I would like to propose the following: > > The Qt Project is actually in favor of changing at least the redirection > > default to exactly what we want it to be. > > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, and > > got approval both by Lars and one of the maintainers. It is merely stuck > > in dealing with the unit test fallout in Qt's CI that somebody has to > > take care of. Doesn't immediately help us of course, but claiming Qt is > > unwilling to change this is simply wrong. > > > > > 1) That effective immediately, QNetworkAccessManager and it's children > > > classes be banned and prohibited within KDE software, to be enforced > > > by server side hooks. > > > 2) That we fork QNetworkAccessManager and the associated classes > > > within the appropriate Framework (to be determined), where the > > > defective behaviour can then be corrected. > > > > While this might solve the redirection problem, it brings us yet another > > then unmaintained HTTP implementation next to the one in KIO, which is a > > far bigger problem long term. We need less of those, not more. > > > > To address the problem of people not using the correct defaults, I did > > propose a much less extreme approach in the above mentioned talk at > > Akademy, namely having a KNetworkAccessManager as a sub-class of QNAM in > > a low-tier frameworks that essentially just enables redirects and HSTS. > > QNAM's implementation isn't the problem after all, the defaults are. > > > > Commit hooks warning about QNAM usage might still be a good idea, but as a > > warning only. Hard blocking QNAM-using code would block at least three > > repos I spend a lot of time on currently, none of which even talks to KDE > > servers. > > > > If you need to take more drastic measures against code not doing this > > correctly regardless, you can do that my dropping the server-side > > workarounds. Yes, that would break the affected application possibly, but > > at least it would not cause massive collateral damage for the people > > using this correctly. > > > > It would also help to know where specifically we have that problem, so we > > can actually solve it, and so we can figure out why we failed to fix this > > there earlier. > > Just bringing this up again - it seems we've not had much movement on > this aside from the Wiki page. > > Examining that Qt merge request, it not only is targeted at just Qt > 6.x, it also hasn't had any movement since the start of October 2019 - > 4 months ago. > Given that we are going to be on Qt 5.x for some time, that merge > request can't be considered to be the 'fix' for this issue. The targeting of Qt6 is due to this aiming at dev when that was still Qt 5.x, but you are right of course, somebody needs to do the work there to drive this to completion, no matter in which version it will actually land. > We need a solution that will ensure software released today doesn't > cause us pain in several years time, because as I illustrated in an > earlier email to this thread, software has a habit of remaining in use > for an extremely long amount of time (August 2014 being the release > date of the Marble versions in question hitting that old URL). > > The easiest path forward is to simply ban QNetworkAccessManager. That might be the easiest path for you, but certainly not for me, given two of the modules I work on most atm are using QNAM (not even to talk to KDE infrastructure), and would suddenly no longer be allowed to be hosted here? Also, I have yet to see a suggestion for a viable alternative to QNAM. The initially proposed fork would mean trading the possibility of broken redirect handling for an unmaintained HTTP stack, I don't think that's a good deal for the security of our users. Instead, how about the way we approached this for KUserFeedback? Before getting things deployed on KDE infrastructure for use by library or application code, sysadmins and developers review if the implementation is allowing easy and secure operation of the infrastructure, redirect handling being one part of this. In case of KUserFeedback you suggest
