Re: Critical Denial of Service bugs in Discover
Hi, Am Samstag, 5. Februar 2022, 22:16:28 CET schrieb Ben Cooksley: > Hi all, > > Over the past week or so Sysadmin has been dealing with an extremely high > volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > This traffic volume is curious in so far that it is directed at two paths > specifically: > - distribute.kde.org/khotnewstuff/fonts-providers.xml > - download.kde.org/ocs/providers.xml > > The first path is an "internal only" host which we were redirecting a > legacy path to prior to the resource being relocated to cdn.kde.org. The > second path has been legacy for numerous years now (more than 5) and is > replaced by autoconfig.kde.org. > It is of extreme concern that these paths are still in use - especially the > ocs/providers.xml one. > >... > > This indicates that the bug lies solely within Plasma's Discover component > - more precisely it's updater. > > Examining the origin of these requests has indicated that some clients are > making requests to these paths well in excess of several times a minute > with a number of IP addresses appearing more 60 times in a 1 minute sized > sample window. FWICT, this is caused by plasma-discover-update, which is triggered by the DiscoverNotifier service if automatic updates are enabled in kcm_updates, updates are available and the system idle for >=15min. // If the system is untouched for 1 hour, trigger the unattened update using namespace std::chrono_literals; KIdleTime::instance()->addIdleTimeout(int(std::chrono::milliseconds(15min).count())); (I wonder whether there's a bug about calling addIdleTimeout more than once. It will then invoke triggerUpdate multiple times after 15min of idle.) The Discover KNS backend creates instances for all available knsrc files, which on construction call KNSReviews::setProviderUrl with the URL defined in those files, triggering the requests. The first URL is used by kfontinst.knsrc from plasma-workspace: ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml The second URL is used by multiple knsrc files in my VM: aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml wallpaperplugin.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > Given that Sysadmin has raised issues with this component and it's > behaviour in the past, it appears that issues regarding the behaviour of > the OCS componentry within Discover remain unresolved. > > Due to the level of distress this is causing our systems, I am therefore > left with no other option other than to direct the Plasma Discover > developers to create and release without delay patches for all versions in > support, as well as for all those currently present in any actively > maintained distributions, that disable all OCS functionality in the > Discover updater. Distributions are requested to treat these patches as > security patches and to distribute them to users without delay. Emergency workarounds for distributions might be to either not ship the KNS backend by not building kns-backend.so or deleting it afterwards, or disabling the discover notifier (/etc/xdg/autostart/org.kde.discover.notifier.desktop) completely. Cheers, Fabian
Re: Critical Denial of Service bugs in Discover
Moin, Am Sonntag, 6. Februar 2022, 19:27:11 CET schrieb Ben Cooksley: > On Sun, Feb 6, 2022 at 1:07 PM Fabian Vogt wrote: > > The first URL is used by kfontinst.knsrc from plasma-workspace: > > ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml > > > > The second URL is used by multiple knsrc files in my VM: > > aurorae.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > comic.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwineffect.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwinscripts.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > kwinswitcher.knsrc:ProvidersUrl=https://download.kde.org/ocs/providers.xml > > wallpaperplugin.knsrc:ProvidersUrl= > > https://download.kde.org/ocs/providers.xml > > This makes me incredibly sad. We had a push to eliminate all usage of the > legacy download.kde.org endpoint many years ago... > I have now resolved the majority of these - if distributions could please > pick up those patches that would be appreciated. > > Please note that I have now terminated the support on the server that was > making these legacy endpoints work, so those patches are necessary to > restore functionality. Does this decrease the server load noticably? On IRC you wrote that the primary offender is "KNewStuff/5.86.0-plasma-discover-update/", can you make the endpoints return an error for the top entries only? Then we'll get bug reports only for the likely cause(s) of the high traffic instead of everyone. What might help is to have a lightweight proxy in front of Apache to handle those paths in a way that doesn't stress the system much. That should probably be investigated independently of the client side, as this is entirely under our control and has immediate effects. It's also possible that the requests aren't actually caused by Discover at all, but just something which imitates it in a DDoS attack. In that case we couldn't do anything on the client-side anyway. I don't think this is very likely, but until the issue was reproduced with disover it's a possibility. > > > Given that Sysadmin has raised issues with this component and it's > > > behaviour in the past, it appears that issues regarding the behaviour of > > > the OCS componentry within Discover remain unresolved. > > > > > > Due to the level of distress this is causing our systems, I am therefore > > > left with no other option other than to direct the Plasma Discover > > > developers to create and release without delay patches for all versions > > in > > > support, as well as for all those currently present in any actively > > > maintained distributions, that disable all OCS functionality in the > > > Discover updater. Distributions are requested to treat these patches as > > > security patches and to distribute them to users without delay. > > > > Emergency workarounds for distributions might be to either not ship the KNS > > backend by not building kns-backend.so or deleting it afterwards, or > > disabling > > the discover notifier > > (/etc/xdg/autostart/org.kde.discover.notifier.desktop) > > completely. > > I have now committed patches to Discover going back to Plasma/5.18 which > disable the build of the KNS backend. > If distributions could please pick them up and distribute them as I > previously indicated that would be much appreciated. > > https://invent.kde.org/plasma/discover/-/commit/f66df3531670592960167f5060feeed6d6c792be IMO we need a more targeted approach there. The main offenders aren't running the latest version, so likely won't get those updates that quickly either. If we have more data and can pinpoint it a bit better that would at least help to speed patch delivery up. > Please note that I intend to investigate whether it is possible to serve > corrupted files from the server side that cause Discover to crash to help > alleviate the load being created by those clients. Sounds like a good way to DoS bugzilla instead and cause bad PR, both up- and downstream. On top of that, it's possible that a resulting crashloop causes an even higher frequency of requests. Cheers, Fabian > Current load being generated by this is: > > 789 requests/sec - 6.4 MB/second - 8.3 kB/request - 1.70113 > ms/request > 217 requests currently being processed, 183 idle workers > > > Cheers, > > Fabian > > > Thanks, > Ben
[plasma/plasma-workspace/Plasma/5.24] kcms/kfontinst/kcmfontinst: Use the correct endpoint for this - hosted by a CDN and therefore capable of handling much greater volumes of traffic.
Git commit fb5656eaf2e021e6a9288edd00573c14afe6e115 by Fabian Vogt, on behalf of Ben Cooksley. Committed on 07/02/2022 at 11:09. Pushed by fvogt into branch 'Plasma/5.24'. Use the correct endpoint for this - hosted by a CDN and therefore capable of handling much greater volumes of traffic. CCMAIL: plasma-devel@kde.org CCMAIL: distributi...@kde.org (cherry picked from commit 8c11f207e2bbf0f98488de7066bd3698705f20e1) M +1-1kcms/kfontinst/kcmfontinst/kfontinst.knsrc https://invent.kde.org/plasma/plasma-workspace/commit/fb5656eaf2e021e6a9288edd00573c14afe6e115 diff --git a/kcms/kfontinst/kcmfontinst/kfontinst.knsrc b/kcms/kfontinst/kcmfontinst/kfontinst.knsrc index 99c195423..fd41b1f89 100644 --- a/kcms/kfontinst/kcmfontinst/kfontinst.knsrc +++ b/kcms/kfontinst/kcmfontinst/kfontinst.knsrc @@ -42,7 +42,7 @@ Name[vi]=Phông chữ Name[x-test]=xxFontsxx Name[zh_CN]=字体 -ProvidersUrl=https://distribute.kde.org/khotnewstuff/fonts-providers.xml +ProvidersUrl=https://cdn.kde.org/khotnewstuff/v1/fonts-providers.xml Categories=KDE-Look.org Fonts TargetDir=kfontinst Uncompress=archive
Re: plasma 5.24 tars ready for packaging
I've published updates to kwin, plasma-workspace and kdeplasma-addons ahead of planned release of Plasma 5.24 tomorrow. Current commit numbers and sha256 list at http://embra.edinburghlinux.co.uk/~jr/tmp/5.24.0-release-data Jonathan On Thu, 3 Feb 2022 at 18:51, Jonathan Riddell wrote: > The tars for Plasma 5.24 are ready on deino for packaging in > distributions. Release is due next Tuesday. > > Jonathan > >
Re: plasma 5.24 tars ready for packaging
I'm not going to publish updates that just remove an important feature. Rather there needs to be discussion in the normal KDE method and that feature should be fixed. Jonathan On Sun, 6 Feb 2022 at 18:46, Ben Cooksley wrote: > On Fri, Feb 4, 2022 at 7:52 AM Jonathan Riddell wrote: > >> The tars for Plasma 5.24 are ready on deino for packaging in >> distributions. Release is due next Tuesday. >> > > Hi Jonathan, > > I've now withdrawn these tarballs as they contain code that performs a > denial of service attack on KDE.org infrastructure. > > As this affects more than just Discover (with KWin, plasma-workspace and > kdeplasma-addons all containing defects that are part of this series as > well) a full respin of all packages will be required. > > We also need patch releases of Discover for all versions going back to > Plasma/5.18. While I appreciate that some of these are "out of support" the > extraordinary nature of the problem we are facing requires it to be made > (much like how Microsoft released a fix for Windows XP in the wake of > Wannacry) > > >> >> Jonathan >> >> > Thanks, > Ben >
kwin respin request
Hi, I'd like to request a re-spin of kwin. These two commits need to be in the final 5.24.0 release. - https://invent.kde.org/plasma/kwin/-/commit/e26ca3d8d4138f6f174c4c0b798b3b7189383d0f - https://invent.kde.org/plasma/kwin/-/commit/9e9bb6c6deaf76834340b9359d0e19fc7ccee8cd Regard, Vlad
Re: plasma 5.24 tars ready for packaging
A second update to KWin tar for Plasma 5.24 kwin;Plasma/5.24;9e9bb6c6deaf76834340b9359d0e19fc7ccee8cd;kwin-5.24.0.tar.xz;c6aec516954c1149aba394530a52c0514b60b21f4d834f58603d648be1bd05a7 On Mon, 7 Feb 2022 at 12:11, Jonathan Riddell wrote: > I've published updates to kwin, plasma-workspace and kdeplasma-addons > ahead of planned release of Plasma 5.24 tomorrow. > > Current commit numbers and sha256 list at > http://embra.edinburghlinux.co.uk/~jr/tmp/5.24.0-release-data > > Jonathan > > > On Thu, 3 Feb 2022 at 18:51, Jonathan Riddell wrote: > >> The tars for Plasma 5.24 are ready on deino for packaging in >> distributions. Release is due next Tuesday. >> >> Jonathan >> >>
Re: plasma 5.24 tars ready for packaging
I should note that plasma-phone-componets had a name change to plasma-mobile. Following discussion with packagers on chat the tars are called plasma-mobile but there are symlinks on the server for the old names. This was done post repo freeze, post beta and without any discussion, it would be really appreciated if developers didn't make this sort of change when the release schedule doesn't allow it. Jonathan On Thu, 3 Feb 2022 at 18:51, Jonathan Riddell wrote: > The tars for Plasma 5.24 are ready on deino for packaging in > distributions. Release is due next Tuesday. > > Jonathan > >
Re: Critical Denial of Service bugs in Discover
On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley wrote: > > Hi all, > > Over the past week or so Sysadmin has been dealing with an extremely high > volume of traffic directed towards both download.kde.org and > distribute.kde.org. > > This traffic volume is curious in so far that it is directed at two paths > specifically: > - distribute.kde.org/khotnewstuff/fonts-providers.xml > - download.kde.org/ocs/providers.xml > > The first path is an "internal only" host which we were redirecting a legacy > path to prior to the resource being relocated to cdn.kde.org. The second path > has been legacy for numerous years now (more than 5) and is replaced by > autoconfig.kde.org. > It is of extreme concern that these paths are still in use - especially the > ocs/providers.xml one. > > The volume of traffic has reached an extent that to prevent the server disk > filling up we have had to disable logging for those two sites. Whilst > dependent on the time of day the server is currently dealing with the current > volume of requests, which is far outside normal specifications: > > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199 > ms/request > > Analysis of a fragment of logs (comprising just a few minutes of traffic) > reveals the following: > > 63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.86.0-plasma-discover-update/" >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0" > > 86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-" > "Mozilla/5.0" > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.5" > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.89.0-discoverupdate/5.23.4" > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.88.0-discoverupdate/5.23.5" > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.90" >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.86.0-plasma-discover-update/" >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "KNewStuff/5.90.0-discoverupdate/5.23.5" >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-" > "Mozilla/5.0" > > This indicates that the bug lies solely within Plasma's Discover component - > more precisely it's updater. > > Examining the origin of these requests has indicated that some clients are > making requests to these paths well in excess of several times a minute with > a number of IP addresses appearing more 60 times in a 1 minute sized sample > window. > > Given that Sysadmin has raised issues with this component and it's behaviour > in the past, it appears that issues regarding the behaviour of the OCS > componentry within Discover remain unresolved. > > Due to the level of distress this is causing our systems, I am therefore left > with no other option other than to direct the Plasma Discover developers to > create and release without delay patches for all versions in support, as well > as for all those currently present in any actively maintained distributions, > that disable all OCS functionality in the Discover updater. Distributions are > requested to treat these patches as security patches and to distribute them > to users without delay. > > In 24 hours time Sysadmin will be making a posting to kde-announce requesting > that users immediately cease use of the Discover update client as it is > creating a Denial of Service attack on our infrastructure. I feel like your response here is out of proportion. Last time we had this conversation, my impression was that the problem was addressed for the most part. If you wanted people working on KNewStuff, Attica or OCS to take any actions, we needed to at the very least have information about you are complaining before you burst out into mailing lists and the likes. In terms of actual solutions this in would probably help to some extent. We never merged it because it's not great design but good results is more important than good design. At the moment they're in their way in but it will take time until it hits users. https://invent.kde.org/frameworks/knewstuff/-/merge_requests/141/ https://invent.kde.org/plasma/discover/-/merge_requests/165/ These were of course not the only mitigation solutions put into place back then. In fact many of them were geared towards giv
Monday meeting notes from 7/2/2022
Nico * Don't crash plasma-integration on invalid color scheme setting: https://invent.kde.org/plasma/plasma-integration/-/merge_requests/34 * Set proper name in desktop file for keditfiletype: https://invent.kde.org/plasma/kde-cli-tools/-/merge_requests/33 * Work around a quirk in macOS in kidletime: https://invent.kde.org/frameworks/kidletime/-/merge_requests/15 * Fix font change notification in qqc2-desktop-style/qqc2-breeze-style: https://invent.kde.org/frameworks/qqc2-desktop-style/-/merge_requests/124 * perf optimization in qqc2-desktop-style/qqc2-breeze-style: https://invent.kde.org/frameworks/qqc2-desktop-style/-/merge_requests/125 * Fix opening files on remote shares in Dolphin: https://invent.kde.org/system/dolphin/-/merge_requests/343 * A question would be whether to backport the perf improvements in qqc2-breeze-style to 5.24 * They are not strictly bug fixess * But perf improvements are very welcome on mobile [ Discussion seems to say yes go ahead] Nate * Did some MRs and lots of bug triage and minor bugfixes for Plasma 5.24 * we have 6 regressions left, if folks wanna have a look: https://bugs.kde.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=ASSIGNED&bug_status=REOPENED&keywords=regression%2C%20&keywords_type=allwords&list_id=1974758&query_format=advanced&version=5.23.90 Arjen * I did some system monitor bugfixing * and also apparently broke a few things in kirigami with the qmlmodule port * mostly because of random path issues * I'm now working on an ecm module that helps with writing qml unit tests, including having a function that adds a very simple instantiation test for a number of items * so that we can quickly verify that "public" qml items still work Fabian * There's this discover ocs topic... * The URL changes we have in 5.24 are at best a workaround, and probably won't hit the actually affected systems soon [apol well I know Ben is angry, I saw taht indeed there were some providers that were not hitting the cache and Ben addressed that] [apol I see there's a whole lot of people using unattended updates which is fairly new and is having an impact there, the changes we are merging today should have an impact] [apol I should look at how it scales for people with a lot of KNS resources that need updating on each run too] [fvogt Would be great if someone could reproduce the issue locally somehow If it's actually really a bug and not just missing optimization] [nicofee There was a discussion about high requests for the providers a while ago, but it seems to have exploded the last two? days Any idea why that could be?] [apol maybe a distro enabled unattended updates by default?] [fvogt I suspect that there's some loop with the updater, e.g. there's an update which fails to download, so it tries in a loop] Marco # Plasma * investigated into https://bugs.kde.org/show_bug.cgi?id=448609 : It is clearly *not* a regreession but something that always happened even in kde4 times every single time the appletsrc file is deleted. All it can be done is a workaround in doing a cleanup of old actions every time plasma starts from the default layout, tough all kglobalaccess api that could do such manipulation is deprecated * A small fix on kcms which have an header (merged): https://invent.kde.org/frameworks/kdeclarative/-/merge_requests/110 * Helping a bit on the power kcm * Final work on the panel keyboard navigation mrs and now all merged \o/ * More work on the plasmashell ScreenPool refactor * please review: https://invent.kde.org/plasma/plasma-workspace/-/merge_requests/1400 * Now ShellCorona is ported ScreenPool as its single source of truth about what screens are available, and primarychanges. Concepts like fake screens or redundant screens are completely hidden from corona. a redundant screen is like it doesn't exists at all for corona * now has a fully working wayland based autotest for ScreenPool work/mart/screenPoolLogicWaylandTest, 14 tests so far which should be quite exaustive (maybe some day will be possible autotest a shellcorona instance itself, even tough that looks even more complicated) * Since is a lot of code copy from Qt's QWayland autotests, i am not completely sure on what to do with that, both on a licensing pow (gpl3 only but since is just in an autotest should be fine) and on a code reusing pow (also, uses some private qplatform api) . perhaps a library exported from kwayland-server?
Re: plasma 5.24 tars ready for packaging
On Tue, Feb 8, 2022 at 1:12 AM Jonathan Riddell wrote: > I'm not going to publish updates that just remove an important feature. > Rather there needs to be discussion in the normal KDE method and that > feature should be fixed. > Sorry but i'm going to categorically reject in the strongest possible terms the above statement. What you are in essence saying is that your view is that it is acceptable to conduct a distributed denial of service attack on someone (even if it unintentional) and then refuse to disable the functionality in question while the issue is investigated in full and fixed properly. That quite simply is appalling. > Jonathan > Regards, Ben > > > On Sun, 6 Feb 2022 at 18:46, Ben Cooksley wrote: > >> On Fri, Feb 4, 2022 at 7:52 AM Jonathan Riddell wrote: >> >>> The tars for Plasma 5.24 are ready on deino for packaging in >>> distributions. Release is due next Tuesday. >>> >> >> Hi Jonathan, >> >> I've now withdrawn these tarballs as they contain code that performs a >> denial of service attack on KDE.org infrastructure. >> >> As this affects more than just Discover (with KWin, plasma-workspace and >> kdeplasma-addons all containing defects that are part of this series as >> well) a full respin of all packages will be required. >> >> We also need patch releases of Discover for all versions going back to >> Plasma/5.18. While I appreciate that some of these are "out of support" the >> extraordinary nature of the problem we are facing requires it to be made >> (much like how Microsoft released a fix for Windows XP in the wake of >> Wannacry) >> >> >>> >>> Jonathan >>> >>> >> Thanks, >> Ben >> >
