Re: Critical Denial of Service bugs in Discover

[Ben Cooksley]

On Tue, Feb 8, 2022 at 4:24 AM Aleix Pol  wrote:

> On Sat, Feb 5, 2022 at 10:16 PM Ben Cooksley  wrote:
> >
> > Hi all,
> >
> > Over the past week or so Sysadmin has been dealing with an extremely
> high volume of traffic directed towards both download.kde.org and
> distribute.kde.org.
> >
> > This traffic volume is curious in so far that it is directed at two
> paths specifically:
> > - distribute.kde.org/khotnewstuff/fonts-providers.xml
> > - download.kde.org/ocs/providers.xml
> >
> > The first path is an "internal only" host which we were redirecting a
> legacy path to prior to the resource being relocated to cdn.kde.org. The
> second path has been legacy for numerous years now (more than 5) and is
> replaced by autoconfig.kde.org.
> > It is of extreme concern that these paths are still in use - especially
> the ocs/providers.xml one.
> >
> > The volume of traffic has reached an extent that to prevent the server
> disk filling up we have had to disable logging for those two sites. Whilst
> dependent on the time of day the server is currently dealing with the
> current volume of requests, which is far outside normal specifications:
> >
> > 555 requests/sec - 4.5 MB/second - 8.3 kB/request - .739199
> ms/request
> >
> > Analysis of a fragment of logs (comprising just a few minutes of
> traffic) reveals the following:
> >
> >  63 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.5"
> >  64 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.4"
> > 104 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.90"
> > 105 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.88.0-discoverupdate/5.23.5"
> >1169 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.86.0-plasma-discover-update/"
> >1256 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.5"
> >2905 "GET /ocs/providers.xml HTTP/1.1" 301 6585 "-" "Mozilla/5.0"
> >
> >  86 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 200 6773 "-"
> "Mozilla/5.0"
> > 130 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.5"
> > 136 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.89.0-discoverupdate/5.23.4"
> > 197 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.88.0-discoverupdate/5.23.5"
> > 199 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.90"
> >2624 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.86.0-plasma-discover-update/"
> >2642 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "KNewStuff/5.90.0-discoverupdate/5.23.5"
> >6117 "GET /khotnewstuff/fonts-providers.xml HTTP/1.1" 304 6132 "-"
> "Mozilla/5.0"
> >
> > This indicates that the bug lies solely within Plasma's Discover
> component - more precisely it's updater.
> >
> > Examining the origin of these requests has indicated that some clients
> are making requests to these paths well in excess of several times a minute
> with a number of IP addresses appearing more 60 times in a 1 minute sized
> sample window.
> >
> > Given that Sysadmin has raised issues with this component and it's
> behaviour in the past, it appears that issues regarding the behaviour of
> the OCS componentry within Discover remain unresolved.
> >
> > Due to the level of distress this is causing our systems, I am therefore
> left with no other option other than to direct the Plasma Discover
> developers to create and release without delay patches for all versions in
> support, as well as for all those currently present in any actively
> maintained distributions, that disable all OCS functionality in the
> Discover updater. Distributions are requested to treat these patches as
> security patches and to distribute them to users without delay.
> >
> > In 24 hours time Sysadmin will be making a posting to kde-announce
> requesting that users immediately cease use of the Discover update client
> as it is creating a Denial of Service attack on our infrastructure.
>
> I feel like your response here is out of proportion.
>
> Last time we had this conversation, my impression was that the problem
> was addressed for the most part. If you wanted people working on
> KNewStuff, Attica or OCS to take any actions, we needed to at the very
> least have information about you are complaining before you burst out
> into mailing lists and the likes.
>

Based on the information I had to hand back in September it was solved yes.
Our server monitoring system indicated that this issue did not exist back
in September - so this is new, although in the same Discover code.


>
> In terms of actual solutions this in would probably help to some
> extent. We never merged it because it's not great design but good
> results is more important than 

Re: plasma-phone-components release process

[Aleix Pol]

On Tue, Feb 8, 2022 at 3:21 PM Jonathan Riddell  wrote:
>
> plasma-phone-components is released as part of Plasma which has an 
> established release process that we review at the start of each cycle.  This 
> time it was renamed after the repo freeze and after the beta which causes 
> unpredictable problems in the packaging process.
>
> Should plasma-phone-components/plasma-mobile continue to be part of the 
> Plasma release? Or would it be better to move it to Plasma Mobile releases?

plasma-mobile is what plasma-phone-components used to be,
plasma-mobile should be released with plasma.

plasma-mobile (and previously plasma-phone-components) is tightly
coupled with certain changes in Plasma and they should go hand in
hand.

Aleix

Re: plasma 5.24 tars ready for packaging

[Nate Graham]

Much work is currently in progress to actually fix these issues. I see 
multiple merge requests across multiple repos being reviewed and merged. 
I think it makes sense to let that process happen. I see no indication 
of the issue not being taken seriously, even considering the hyperbolic 
and threatening way in which it was communicated mere days before a 
major software release that is already occupying everyone's time. Let's 
tone down the rhetoric and let developers do their jobs, now that 
they've been made aware of this critical issue.


Nate


On 2/8/22 02:53, Jonathan Riddell wrote:
You'll need to take this up with the maintainers of Discover and 
KNewStuff.  There's no reason why fixing the issue wouldn't resolve the 
problem as fast as removing it.


Jonathan


On Tue, 8 Feb 2022 at 06:53, Ben Cooksley > wrote:


On Tue, Feb 8, 2022 at 1:12 AM Jonathan Riddell mailto:j...@jriddell.org>> wrote:

I'm not going to publish updates that just remove an important
feature.  Rather there needs to be discussion in the normal KDE
method and that feature should be fixed.


Sorry but i'm going to categorically reject in the strongest
possible terms the above statement.

What you are in essence saying is that your view is that it is
acceptable to conduct a distributed denial of service attack on
someone (even if it unintentional) and then refuse to disable the
functionality in question while the issue is investigated in full
and fixed properly.
That quite simply is appalling.


Jonathan


Regards,
Ben



On Sun, 6 Feb 2022 at 18:46, Ben Cooksley mailto:bcooks...@kde.org>> wrote:

On Fri, Feb 4, 2022 at 7:52 AM Jonathan Riddell
mailto:j...@jriddell.org>> wrote:

The tars for Plasma 5.24 are ready on deino for
packaging in distributions.  Release is due next Tuesday.


Hi Jonathan,

I've now withdrawn these tarballs as they contain code that
performs a denial of service attack on KDE.org infrastructure.

As this affects more than just Discover (with KWin,
plasma-workspace and kdeplasma-addons all containing defects
that are part of this series as well) a full respin of all
packages will be required.

We also need patch releases of Discover for all versions
going back to Plasma/5.18. While I appreciate that some of
these are "out of support" the extraordinary nature of the
problem we are facing requires it to be made (much like how
Microsoft released a fix for Windows XP in the wake of Wannacry)


Jonathan


Thanks,
Ben

plasma-phone-components release process

[Jonathan Riddell]

plasma-phone-components is released as part of Plasma which has an
established release process that we review at the start of each cycle.
This time it was renamed after the repo freeze and after the beta which
causes unpredictable problems in the packaging process.

Should plasma-phone-components/plasma-mobile continue to be part of the
Plasma release? Or would it be better to move it to Plasma Mobile releases?

Jonathan

Plasma 5.24.0 LTS

[Jonathan Riddell]

Plasma 5.24.0 LTS is now released
https://kde.org/announcements/plasma/5/5.24.0

Re: plasma 5.24 tars ready for packaging

[Jonathan Riddell]

You'll need to take this up with the maintainers of Discover and
KNewStuff.  There's no reason why fixing the issue wouldn't resolve the
problem as fast as removing it.

Jonathan


On Tue, 8 Feb 2022 at 06:53, Ben Cooksley  wrote:

> On Tue, Feb 8, 2022 at 1:12 AM Jonathan Riddell  wrote:
>
>> I'm not going to publish updates that just remove an important feature.
>> Rather there needs to be discussion in the normal KDE method and that
>> feature should be fixed.
>>
>
> Sorry but i'm going to categorically reject in the strongest possible
> terms the above statement.
>
> What you are in essence saying is that your view is that it is acceptable
> to conduct a distributed denial of service attack on someone (even if it
> unintentional) and then refuse to disable the functionality in question
> while the issue is investigated in full and fixed properly.
> That quite simply is appalling.
>
>
>> Jonathan
>>
>
> Regards,
> Ben
>
>
>>
>>
>> On Sun, 6 Feb 2022 at 18:46, Ben Cooksley  wrote:
>>
>>> On Fri, Feb 4, 2022 at 7:52 AM Jonathan Riddell  wrote:
>>>
 The tars for Plasma 5.24 are ready on deino for packaging in
 distributions.  Release is due next Tuesday.

>>>
>>> Hi Jonathan,
>>>
>>> I've now withdrawn these tarballs as they contain code that performs a
>>> denial of service attack on KDE.org infrastructure.
>>>
>>> As this affects more than just Discover (with KWin, plasma-workspace and
>>> kdeplasma-addons all containing defects that are part of this series as
>>> well) a full respin of all packages will be required.
>>>
>>> We also need patch releases of Discover for all versions going back to
>>> Plasma/5.18. While I appreciate that some of these are "out of support" the
>>> extraordinary nature of the problem we are facing requires it to be made
>>> (much like how Microsoft released a fix for Windows XP in the wake of
>>> Wannacry)
>>>
>>>

 Jonathan


>>> Thanks,
>>> Ben
>>>
>>

Re: plasma 5.24 tars ready for packaging

[Jonathan Riddell]

A late update to Discover 5.24 tar today

discover;Plasma/5.24;84d59c60eaf4714681db0f38a935ba03e9e5f019;discover-5.24.0.tar.xz;18889fa78d00dec8b112e09a177aa286079e5946cb2f9d15c43c0c6515de6e35


On Thu, 3 Feb 2022 at 18:51, Jonathan Riddell  wrote:

> The tars for Plasma 5.24 are ready on deino for packaging in
> distributions.  Release is due next Tuesday.
>
> Jonathan
>
>

Re: Monday meeting notes from 7/2/2022

[Ben Cooksley]

On Tue, Feb 8, 2022 at 4:43 AM Marco Martin  wrote:

> Nico
> * Don't crash plasma-integration on invalid color scheme setting:
> https://invent.kde.org/plasma/plasma-integration/-/merge_requests/34
> * Set proper name in desktop file for keditfiletype:
> https://invent.kde.org/plasma/kde-cli-tools/-/merge_requests/33
> * Work around a quirk in macOS in kidletime:
> https://invent.kde.org/frameworks/kidletime/-/merge_requests/15
> * Fix font change notification in
> qqc2-desktop-style/qqc2-breeze-style:
> https://invent.kde.org/frameworks/qqc2-desktop-style/-/merge_requests/124
> * perf optimization in qqc2-desktop-style/qqc2-breeze-style:
> https://invent.kde.org/frameworks/qqc2-desktop-style/-/merge_requests/125
> * Fix opening files on remote shares in Dolphin:
> https://invent.kde.org/system/dolphin/-/merge_requests/343
> * A question would be whether to backport the perf improvements in
> qqc2-breeze-style to 5.24
> * They are not strictly bug fixess
> * But perf improvements are very welcome on mobile
> [ Discussion seems to say yes go ahead]
>
> Nate
> * Did some MRs and lots of bug triage and minor bugfixes for  Plasma 5.24
> * we have 6 regressions left, if folks wanna have a look:
>
> https://bugs.kde.org/buglist.cgi?bug_status=UNCONFIRMED_status=CONFIRMED_status=ASSIGNED_status=REOPENED=regression%2C%20_type=allwords_id=1974758_format=advanced=5.23.90
>
> Arjen
> * I did some system monitor bugfixing
> * and also apparently broke a few things in kirigami with the qmlmodule
> port
> * mostly because of random path issues
> * I'm now working on an ecm module that helps with writing qml unit
> tests, including having a function that adds a very simple
> instantiation test for a number of items
> * so that we can quickly verify that "public" qml items still work
>
> Fabian
> * There's this discover ocs topic...
> * The URL changes we have in 5.24 are at best a workaround, and
> probably won't hit the actually affected systems soon
> [apol well I know Ben is angry, I saw taht indeed there were some
> providers that were not hitting the cache and Ben addressed that]
>

Hopefully I got them all - unfortunately as I don't have everything
installed locally there is no guarantee that they have all been fixed.
I should note that LXR does not appear to pick up *.knsrc files too well,
so it cannot be relied on either.

This workaround should not in any form be considered a proper fix for the
issue.


> [apol I see there's a whole lot of people using unattended updates
> which is fairly new and is having an impact there, the changes we are
> merging today should have an impact]
> [apol I should look at how it scales for people with a lot of KNS
> resources that need updating on each run too]
>

Based on the traffic we are seeing, it appears that Discover makes a
request for each KNS resource the user has locally, which is fairly
non-scalable.


> [fvogt Would be great if someone could reproduce the issue locally
> somehow If it's actually really a bug and not just missing
> optimization]
> [nicofee There was a discussion about high requests for the providers
> a while ago, but it seems to have exploded the last two? days  Any
> idea why that could be?]
>

It crossed over a crucial threshold to come to Sysadmin's attention two
days ago, however our server monitoring system indicates that the base load
on the system has been climbing since at least October 2021 (we don't have
earlier data, however the load I am seeing in October is higher than what I
recall the base system load as being when it was originally deployed - it
should be around 1 during the EU day time and sub 1 during EU night).

This trend would be consistent with the KF 5.86 based distributions
starting to ship the automatic updater client.

I should note that high traffic load on autoconfig.kde.org led Sysadmin to
move autoconfig.kde.org to being CDN based back in September, so the issue
predates the release of Ubuntu 20.10.
This was when the issue was originally raised and when changes were made to
start setting User Agents and some caching was added (although seemingly
not enough?)

Please note that we need to backport any fixes and ask/firmly
push distributions to roll them out to users - preferably as security fixes
- otherwise it will take many months if not years for this situation to be
resolved.
This should be done even for versions we consider "out of support" - if an
actively current/supported distribution releases uses that version we
should ship an update.


> [apol maybe a distro enabled unattended updates by default?]
> [fvogt  I suspect that there's some loop with the updater, e.g.
> there's an update which fails to download, so it tries in a loop]
>
> Marco
> # Plasma
> * investigated into https://bugs.kde.org/show_bug.cgi?id=448609 : It
> is clearly *not* a regreession but something that always happened even
> in kde4 times every single time the appletsrc file is deleted. All it
> can be done is a workaround in doing a cleanup