On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley <bcooks...@kde.org> wrote: > > Hi all, > > Over the past week or so Sysadmin has been dealing with an extremely high > volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > This traffic volume is curious in so far that it is directed at two paths > specifically: > - distribute.kde.org/khotnewstuff/fonts-providers.xml > - download.kde.org/ocs/providers.xml > > The first path is an "internal only" host which we were redirecting a legacy > path to prior to the resource being relocated to cdn.kde.org. The second path > has been legacy for numerous years now (more than 5) and is replaced by > autoconfig.kde.org. > It is of extreme concern that these paths are still in use - especially the > ocs/providers.xml one. > > The volume of traffic has reached an extent that to prevent the server disk > filling up we have had to disable logging for those two sites. Whilst > dependent on the time of day the server is currently dealing with the current > volume of requests, which is far outside normal specifications: > > <dt>555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199 > ms/request</dt> > > Analysis of a fragment of logs (comprising just a few minutes of traffic) > reveals the following: > > 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > 1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.86.0-plasma-discover-update/" > 1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" > 2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0" > > 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-" > "Mozilla/5.0" > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > 2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.86.0-plasma-discover-update/" > 2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" > 6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "Mozilla/5.0" > > This indicates that the bug lies solely within Plasma's Discover component - > more precisely it's updater. > > Examining the origin of these requests has indicated that some clients are > making requests to these paths well in excess of several times a minute with > a number of IP addresses appearing more 60 times in a 1 minute sized sample > window. > > Given that Sysadmin has raised issues with this component and it's behaviour > in the past, it appears that issues regarding the behaviour of the OCS > componentry within Discover remain unresolved. > > Due to the level of distress this is causing our systems, I am therefore left > with no other option other than to direct the Plasma Discover developers to > create and release without delay patches for all versions in support, as well > as for all those currently present in any actively maintained distributions, > that disable all OCS functionality in the Discover updater. Distributions are > requested to treat these patches as security patches and to distribute them > to users without delay. > > In 24 hours time Sysadmin will be making a posting to kde-announce requesting > that users immediately cease use of the Discover update client as it is > creating a Denial of Service attack on our infrastructure.
I feel like your response here is out of proportion. Last time we had this conversation, my impression was that the problem was addressed for the most part. If you wanted people working on KNewStuff, Attica or OCS to take any actions, we needed to at the very least have information about you are complaining before you burst out into mailing lists and the likes. In terms of actual solutions this in would probably help to some extent. We never merged it because it's not great design but good results is more important than good design. At the moment they're in their way in but it will take time until it hits users. https://invent.kde.org/frameworks/knewstuff/-/merge_requests/141/ https://invent.kde.org/plasma/discover/-/merge_requests/165/ These were of course not the only mitigation solutions put into place back then. In fact many of them were geared towards giving more information about what was happening and I've still to get any feedback there. Without information it's impossible that we can take any solutions. For what it's worth, I looked into what was happening within Qt and indeed using download.kde.org instead of autoconfig.kde.org is not hitting the cache paths. If this is something that worries us, maybe we should put together tools that help us detect this kind of problems as exploding in here does nothing to helpthis issue, only stresses people out. Personally, I found very demotivating how the problem was presented once again. Aleix Aleix
