Re: Signing keys for commercial app stores
On Tue, Jun 11, 2019 at 1:37 AM Aleix Pol wrote: > > On Mon, Jun 10, 2019 at 4:03 AM Simon Redman wrote: > > > > Hello, > > > > I am Simon, and I work on KDE Connect. This summer, KDE Connect has two > > excellent GSoC students, one working on a MacOS port and one working on > > a Windows port, with the end goal of bringing those ports to feature > > pairity with our Linux version and doing an official release. > > > > While we could just post our releases to some X.kde.org website and > > distribute unsigned binaries, this would not reach as many users as > > having them properly signed and released via the offical MacOS and > > Windows app stores. > > > > Does anyone have experience with: > > A. Windows App Store Releases > > B. MacOS App Store Release > > We do have experience on Android. Is the story on Windows/Mac all that > different? Windows is essentially a solved problem. Mac on the other hand isn't really solvable. > > Aleix Cheers, Ben
Re: Signing keys for commercial app stores
On Tue, 11 Jun 2019, 16:48 Jonathan Aquilina, wrote: > Also I forgot to add you need to have a developers subscription to be able > to release on the apple app store which is $99 per year but you can release > as many apps as you want > Hi Jonathan, Thanks for those details. Around 2 years ago we (the KDE e.V) bought an Apple Mac Mini machine so fortunately hardware isn't an issue here. That machine is currently connected to the Binary Factory, and produces nightly unsigned DMG images for various applications. The only part missing is the Apple Developer ID you mentioned (along with notarisation) > Regards > Jonathan > Cheers, Ben > On 11/06/2019, 06:47, "Jonathan Aquilina" > wrote: > > First thigns first you need apple hardware to develop on with their > Xcode ide im sure to work with it in some way some how to compile what you > are working on. > > Also if you don’t have mac hardware you can actually rent mini servers > with full SSH access from hostmyapple.com > > Regards, > Jonathan > > On 10/06/2019, 20:41, "kde-devel on behalf of Ben Cooksley" < > kde-devel-boun...@kde.org on behalf of bcooks...@kde.org> wrote: > > On Tue, Jun 11, 2019 at 1:37 AM Aleix Pol > wrote: > > > > On Mon, Jun 10, 2019 at 4:03 AM Simon Redman > wrote: > > > > > > Hello, > > > > > > I am Simon, and I work on KDE Connect. This summer, KDE > Connect has two > > > excellent GSoC students, one working on a MacOS port and one > working on > > > a Windows port, with the end goal of bringing those ports to > feature > > > pairity with our Linux version and doing an official release. > > > > > > While we could just post our releases to some X.kde.org > website and > > > distribute unsigned binaries, this would not reach as many > users as > > > having them properly signed and released via the offical MacOS > and > > > Windows app stores. > > > > > > Does anyone have experience with: > > > A. Windows App Store Releases > > > B. MacOS App Store Release > > > > We do have experience on Android. Is the story on Windows/Mac > all that > > different? > > Windows is essentially a solved problem. Mac on the other hand > isn't > really solvable. > > > > > Aleix > > Cheers, > Ben > > > > >
Re: Signing keys for commercial app stores
On Tue, Jun 11, 2019 at 8:29 PM Jonathan Aquilina wrote: > > Hi Ben i can get a developers account if need be and can push this stuff to > the App Store if it helps out. For those wondering, i've replied to Jonathan off list regarding this. Cheers, Ben > > Regards, > Jonathan > > Get Outlook for iOS > > From: kde-devel on behalf of Ben Cooksley > > Sent: Tuesday, June 11, 2019 10:13:31 AM > To: kde-devel > Subject: Re: Signing keys for commercial app stores > > On Tue, 11 Jun 2019, 16:48 Jonathan Aquilina, wrote: >> >> Also I forgot to add you need to have a developers subscription to be able >> to release on the apple app store which is $99 per year but you can release >> as many apps as you want > > > Hi Jonathan, > > Thanks for those details. > > Around 2 years ago we (the KDE e.V) bought an Apple Mac Mini machine so > fortunately hardware isn't an issue here. > > That machine is currently connected to the Binary Factory, and produces > nightly unsigned DMG images for various applications. > > The only part missing is the Apple Developer ID you mentioned (along with > notarisation) > >> >> Regards >> Jonathan > > > Cheers, > Ben > >> >> On 11/06/2019, 06:47, "Jonathan Aquilina" wrote: >> >> First thigns first you need apple hardware to develop on with their >> Xcode ide im sure to work with it in some way some how to compile what you >> are working on. >> >> Also if you don’t have mac hardware you can actually rent mini servers >> with full SSH access from hostmyapple.com >> >> Regards, >> Jonathan >> >> On 10/06/2019, 20:41, "kde-devel on behalf of Ben Cooksley" >> wrote: >> >> On Tue, Jun 11, 2019 at 1:37 AM Aleix Pol wrote: >> > >> > On Mon, Jun 10, 2019 at 4:03 AM Simon Redman >> wrote: >> > > >> > > Hello, >> > > >> > > I am Simon, and I work on KDE Connect. This summer, KDE Connect >> has two >> > > excellent GSoC students, one working on a MacOS port and one >> working on >> > > a Windows port, with the end goal of bringing those ports to >> feature >> > > pairity with our Linux version and doing an official release. >> > > >> > > While we could just post our releases to some X.kde.org website >> and >> > > distribute unsigned binaries, this would not reach as many users >> as >> > > having them properly signed and released via the offical MacOS >> and >> > > Windows app stores. >> > > >> > > Does anyone have experience with: >> > > A. Windows App Store Releases >> > > B. MacOS App Store Release >> > >> > We do have experience on Android. Is the story on Windows/Mac all >> that >> > different? >> >> Windows is essentially a solved problem. Mac on the other hand isn't >> really solvable. >> >> > >> > Aleix >> >> Cheers, >> Ben >> >> >> >>
Downtime Announcement: Transactional Email Infrastructure
Hi all, We've been informed by one of our hosting providers that they need to schedule some downtime in order to allow for a server move. This will take place this coming Monday, from approximately 4pm UTC for up to 4 hours. As a result, transactional email services will be unavailable during this time meaning email notifications from Bugzilla, Phabricator, Gitlab, Jenkins and other services hosted on KDE Infrastructure will be held and delivered once normal service has resumed. IRC Bots supported by email notifications, including both commits and bugs may also be impacted and will catchup once service resumes as well. Email aliases and mailing lists will not be affected by this downtime, and the services themselves should continue to operate normally. Should anyone have any questions, please let us know. Regards, Ben Cooksley KDE Sysadmin
Issues with Jenkins Builds
Hi all, Recently we were affected by a regression within Jenkins, the effect of which meant that in some circumstances builds would not be triggered when new commits were introduced. In addition, the views showing the list of all builds would also not show the last successfully completed build in some circumstances. This fault has now been worked around on build.kde.org, and rebuilds have been initiated of a number of jobs which have been identified as either affected or potentially affected, which unfortunately will take the better part of the next day or so for the system to process. For all those projects not being rebuilt, maintainers are asked to please check their builds to ensure that the latest build reflects the current state of their repository. Particular attention should be paid to stable builds, which are the most likely to be affected by the above issues. Should your project(s) builds be affected by this issue (with the latest build being an old one) then the issue will be corrected by performing a rebuild manually, after which it should behave properly. Please respond to this email with the names of the affected projects and we'll arrange for this if you don't have access to do this yourself. Apologies for the inconvenience caused. Regards, Ben Cooksley KDE Sysadmin
Eliminating the last CI Failures
Hi all, For a while now we've had a couple of projects which have been persistently failing on all platforms, which it would be nice to get fixed. If someone would like to take a look into one of the following, it would be appreciated, and would help to eliminate the last couple of items from the Failing tab at https://build.kde.org/ - Kajongg: Fails with a CMake error due to the build path getting wrapped up in part of the target name. This probably needs someone with a bit of CMake knowledge to rework how it sets up targets. - Signon KWallet Extension: Always tries to install outside of the prefix specified at build time, which will always fail on the CI system (or lead to situations where your tests fail due to components being missing or inaccessible). The fix for this would be to respect (but warn about) the lack of alignment between the install prefix and the system prefix. There are also issues with Dr Konqi, Umbrello and KStars on Windows (related to exports and linking), while Latte Dock has issues on FreeBSD (something to do with a QMetaObject definition) Any fixes for the above would be much appreciated! Thanks, Ben
Retirement of notes.kde.org
Good evening all, Currently we're in the situation where the software we use to run notes.kde.org is both difficult to maintain, as well as support (for things like document confidentiality for various groups within KDE). We'd therefore like to retire the service, replacing it with the Realtime Text Editor within Nextcloud (share.kde.org). Internally this editor uses Markdown. You can find a demo of this at https://share.kde.org/s/gtFcRmwetRKqTZJ (no login required) It would be appreciated if everyone could please test the realtime text editor and let us know if they encounter any issues. In terms of feature differences, we are aware that highlighting/authorship information won't be retained by the new editor, and there can occasionally be problems when editing the same line with someone else simultaneously. If everyone is okay with this we'd like to go ahead with shutting down notes.kde.org as soon as possible. Thanks, Ben Cooksley KDE Sysadmin
Re: Retirement of notes.kde.org
On Mon, Sep 16, 2019 at 5:14 AM Valorie Zimmerman wrote: > > Hey Ben, Hi Valorie, > > On Sun, Sep 15, 2019 at 11:58 AM Ben Cooksley wrote: >> >> Good evening all, >> >> Currently we're in the situation where the software we use to run >> notes.kde.org is both difficult to maintain, as well as support (for >> things like document confidentiality for various groups within KDE). >> >> We'd therefore like to retire the service, replacing it with the >> Realtime Text Editor within Nextcloud (share.kde.org). Internally this >> editor uses Markdown. >> >> You can find a demo of this at https://share.kde.org/s/gtFcRmwetRKqTZJ >> (no login required) >> >> It would be appreciated if everyone could please test the realtime >> text editor and let us know if they encounter any issues. >> >> In terms of feature differences, we are aware that >> highlighting/authorship information won't be retained by the new >> editor, and there can occasionally be problems when editing the same >> line with someone else simultaneously. >> >> If everyone is okay with this we'd like to go ahead with shutting down >> notes.kde.org as soon as possible. >> >> Thanks, >> Ben Cooksley >> KDE Sysadmin > > > I noticed that most every BoF at Akademy this year used notes to link the BoF > notes. Will all these notes be just disappeared? If so, that is really > unfortunate. There is a reason many of us continue to use notes. One can see > who typed what, one can see the history of the doc - and other points > mentioned on https://share.kde.org/s/gtFcRmwetRKqTZJ. > > ::sigh:: The content of notes.kde.org will be exported - likely in ODT format - prior to us shutting it down. The resulting files will be made available on share.kde.org. If there are things which need to be resolved prior to the shutdown of notes.kde.org we can look into those. While it is unfortunate that the links will be broken, we should be able to make them relatively close (ie. send the user to a public view of the folder on share.kde.org which contains all of the content exported from notes.kde.org) so the user isn't too far from the original link. In terms of feature regressions i'm tracking, I see the following: - Lack of editor highlighting - No stored history aside from Nextcloud versioning - Slightly harder to create new documents Any other comments on this which I missed from the document? Cheers, Ben > > Valorie > -- > http://about.me/valoriez - pronouns: she/her > >
[IMPORTANT] Binary Factory & CI System Maintenance
Hi all, As part of changes to help improve the capacity, capability and security of the Binary Factory and CI system, i'm going to be performing a rebuild of several of the physical hosts responsible for performing builds on the CI system. Following these changes, signing operations for Flatpak and Android builds will have been shifted away from the machines responsible for handling the actual builds, and the system will have an additional machine added to handle builds. This will also pave the way for the build process for Snaps being brought to the Binary Factory (although there are other issues which may hold up the delivery of that due to excessive requirements imposed by the Snap tooling) During this time however, the overall capacity of the system may be reduced and the capability to perform some builds may be unavailable. Should anyone have any upcoming releases that require the services of the Binary Factory & CI system it would be appreciated if you could please let me know so I can take that into account for scheduling the various changes. Thanks, Ben Cooksley KDE Sysadmin
Re: [IMPORTANT] Binary Factory & CI System Maintenance
On Tue, Sep 17, 2019 at 2:57 PM sithlord48 wrote: > > Can we get a process to build appimages? This needs the tooling for it to be built first. Given that Craft already has the majority of the infrastructure for this, i'd suggest adding appimage support to it which should then make adding support to the Binary Factory straight forward and efficient in terms of resource utilisation. Existing work on Appimages has tended to be application specific and therefore not suitable for easy reuse (which Craft would solve) Cheers, Ben > > On Mon, Sep 16, 2019 at 3:47 PM Ben Cooksley wrote: >> >> Hi all, >> >> As part of changes to help improve the capacity, capability and >> security of the Binary Factory and CI system, i'm going to be >> performing a rebuild of several of the physical hosts responsible for >> performing builds on the CI system. >> >> Following these changes, signing operations for Flatpak and Android >> builds will have been shifted away from the machines responsible for >> handling the actual builds, and the system will have an additional >> machine added to handle builds. >> >> This will also pave the way for the build process for Snaps being >> brought to the Binary Factory (although there are other issues which >> may hold up the delivery of that due to excessive requirements imposed >> by the Snap tooling) >> >> During this time however, the overall capacity of the system may be >> reduced and the capability to perform some builds may be unavailable. >> >> Should anyone have any upcoming releases that require the services of >> the Binary Factory & CI system it would be appreciated if you could >> please let me know so I can take that into account for scheduling the >> various changes. >> >> Thanks, >> Ben Cooksley >> KDE Sysadmin
Re: Retirement of notes.kde.org
Hi all, Just bringing this thread to a bit of a conclusion, are we all happy for the replacement of notes.kde.org to go ahead? >From my reading through of this thread, i've seen people raise concerns about the following: - Author colours (which weren't 100% reliable on notes.kde.org in any case as they sometimes disappeared) - History information - Occasional issues with typing on the same line - More complex process to start a new pad or be linked to an existing one >From my understanding, the developers of the plugin we're using (which is under active development) have plans to bring author tracking / colouring in a later release, and chances are there will be further improvements to syncing in later releases as well. Thanks, Ben
Re: Retirement of notes.kde.org
On Thu, Sep 19, 2019 at 2:50 AM Aleix Pol wrote: > > On Sun, Sep 15, 2019 at 8:58 PM Ben Cooksley wrote: > > > > Good evening all, > > > > Currently we're in the situation where the software we use to run > > notes.kde.org is both difficult to maintain, as well as support (for > > things like document confidentiality for various groups within KDE). > > > > We'd therefore like to retire the service, replacing it with the > > Realtime Text Editor within Nextcloud (share.kde.org). Internally this > > editor uses Markdown. > > > > You can find a demo of this at https://share.kde.org/s/gtFcRmwetRKqTZJ > > (no login required) > > > > It would be appreciated if everyone could please test the realtime > > text editor and let us know if they encounter any issues. > > > > In terms of feature differences, we are aware that > > highlighting/authorship information won't be retained by the new > > editor, and there can occasionally be problems when editing the same > > line with someone else simultaneously. > > > > If everyone is okay with this we'd like to go ahead with shutting down > > notes.kde.org as soon as possible. > > > > Thanks, > > Ben Cooksley > > KDE Sysadmin > > Hi, > Is the new editor already usable on share.kde.org? If I create a text > file I get a different thing than your demo. It should be yes. Did you create a new document or a new text document? This new editor only works with text documents (slightly confusingly) as the document editor is a version of LibreOffice Online (for more fully featured documents) > > Aleix Cheers, Ben
Re: Retirement of notes.kde.org
On Thu, Sep 19, 2019 at 12:55 PM Aleix Pol wrote: > > On Thu, Sep 19, 2019 at 4:34 AM Ben Cooksley wrote: > > > > On Thu, Sep 19, 2019 at 2:50 AM Aleix Pol wrote: > > > > > > On Sun, Sep 15, 2019 at 8:58 PM Ben Cooksley wrote: > > > > > > > > Good evening all, > > > > > > > > Currently we're in the situation where the software we use to run > > > > notes.kde.org is both difficult to maintain, as well as support (for > > > > things like document confidentiality for various groups within KDE). > > > > > > > > We'd therefore like to retire the service, replacing it with the > > > > Realtime Text Editor within Nextcloud (share.kde.org). Internally this > > > > editor uses Markdown. > > > > > > > > You can find a demo of this at https://share.kde.org/s/gtFcRmwetRKqTZJ > > > > (no login required) > > > > > > > > It would be appreciated if everyone could please test the realtime > > > > text editor and let us know if they encounter any issues. > > > > > > > > In terms of feature differences, we are aware that > > > > highlighting/authorship information won't be retained by the new > > > > editor, and there can occasionally be problems when editing the same > > > > line with someone else simultaneously. > > > > > > > > If everyone is okay with this we'd like to go ahead with shutting down > > > > notes.kde.org as soon as possible. > > > > > > > > Thanks, > > > > Ben Cooksley > > > > KDE Sysadmin > > > > > > Hi, > > > Is the new editor already usable on share.kde.org? If I create a text > > > file I get a different thing than your demo. > > > > It should be yes. Did you create a new document or a new text document? > > > > This new editor only works with text documents (slightly confusingly) > > as the document editor is a version of LibreOffice Online (for more > > fully featured documents) > > > > > > > > Aleix > > > > Cheers, > > Ben > > It was a new text document, AFAIR. This is what it looks like: > https://i.imgur.com/6RhmNLK.png Ah, that is the same editor, just in logged in mode. If you were to create a publicly accessible link, and use that to access the document then you'd get a view that looks exactly like the one I created. In the logged in version you get the ability to see things like prior versions (accessible through the sidebar, which you can open by clicking the icon in the top right) If you're missing any controls then you may need to perform a force refresh (Ctrl + F5). > > Aleix Cheers, Ben
Re: Retirement of notes.kde.org
On Thu, Sep 19, 2019 at 1:11 PM Aleix Pol wrote: > > On Thu, Sep 19, 2019 at 1:09 PM Ben Cooksley wrote: > > > > On Thu, Sep 19, 2019 at 12:55 PM Aleix Pol wrote: > > > > > > On Thu, Sep 19, 2019 at 4:34 AM Ben Cooksley wrote: > > > > > > > > On Thu, Sep 19, 2019 at 2:50 AM Aleix Pol wrote: > > > > > > > > > > On Sun, Sep 15, 2019 at 8:58 PM Ben Cooksley > > > > > wrote: > > > > > > > > > > > > Good evening all, > > > > > > > > > > > > Currently we're in the situation where the software we use to run > > > > > > notes.kde.org is both difficult to maintain, as well as support (for > > > > > > things like document confidentiality for various groups within KDE). > > > > > > > > > > > > We'd therefore like to retire the service, replacing it with the > > > > > > Realtime Text Editor within Nextcloud (share.kde.org). Internally > > > > > > this > > > > > > editor uses Markdown. > > > > > > > > > > > > You can find a demo of this at > > > > > > https://share.kde.org/s/gtFcRmwetRKqTZJ > > > > > > (no login required) > > > > > > > > > > > > It would be appreciated if everyone could please test the realtime > > > > > > text editor and let us know if they encounter any issues. > > > > > > > > > > > > In terms of feature differences, we are aware that > > > > > > highlighting/authorship information won't be retained by the new > > > > > > editor, and there can occasionally be problems when editing the same > > > > > > line with someone else simultaneously. > > > > > > > > > > > > If everyone is okay with this we'd like to go ahead with shutting > > > > > > down > > > > > > notes.kde.org as soon as possible. > > > > > > > > > > > > Thanks, > > > > > > Ben Cooksley > > > > > > KDE Sysadmin > > > > > > > > > > Hi, > > > > > Is the new editor already usable on share.kde.org? If I create a text > > > > > file I get a different thing than your demo. > > > > > > > > It should be yes. Did you create a new document or a new text document? > > > > > > > > This new editor only works with text documents (slightly confusingly) > > > > as the document editor is a version of LibreOffice Online (for more > > > > fully featured documents) > > > > > > > > > > > > > > Aleix > > > > > > > > Cheers, > > > > Ben > > > > > > It was a new text document, AFAIR. This is what it looks like: > > > https://i.imgur.com/6RhmNLK.png > > > > Ah, that is the same editor, just in logged in mode. > > > > If you were to create a publicly accessible link, and use that to > > access the document then you'd get a view that looks exactly like the > > one I created. > > In the logged in version you get the ability to see things like prior > > versions (accessible through the sidebar, which you can open by > > clicking the icon in the top right) > > > > If you're missing any controls then you may need to perform a force > > refresh (Ctrl + F5). > > > > > > > > Aleix > > > > Cheers, > > Ben > > Ah, okay, cool, thanks! > > BTW, it will be really handy to be able to have publicly editable > texts, this had been a problem for me in the past (that people needed > to be in identity.kde.org). No worries - glad to hear that the feature will be very useful. The ability to have publicly editable texts is one of several feature improvements we get by switching to this new editor :) > > Aleix Cheers, Ben
Replacement of notes.kde.org
Good morning everyone, As previously discussed on the KDE Community mailing list, we've now retired notes.kde.org from service in favour of the Nextcloud realtime editor and it's associated functionality. To assist in making this functionality as easy to work with as possible, Sysadmin has provisioned a community shared folder to host these documents. This folder has been pre-populated with exported copies of the public pads in HTML format which had previously been hosted on notes.kde.org. This folder can be accessed at any time by visiting https://notes.kde.org/p/ or by logging into share.kde.org and selecting the "Community Notes" folder. For those who prefer anonymous/read-only access, that can be achieved by visiting https://notes.kde.org/ (which will then redirect you to the public shared link on share.kde.org) Going forward should you need to create new documents to collaboratively edit, you can continue to do so by creating them in that folder where they'll be accessible by all other members of the community. For those who prefer a direct link to documents, you can find these by opening the document in question (creating it first if necessary) then opening the sidebar, and clicking the copy/clipboard icon on the far right, immediately below the X you would use to close the sidebar. Please note that these links will require the user to login however. Should you need to create a link for anyone to edit a document, please feel free to create a public share link for the document. This can be done by opening the sidebar, then selecting Sharing and selecting the option to create a Share link, ensuring you tick the option to "Allow editing". Please note that in the interests of preventing data-loss we have restricted the folder to prevent files from being deleted or otherwise removed from it. Should you find anything which needs removing, please file a Sysadmin ticket. Should anyone have any questions regarding any of the above, please let us know. Thanks, Ben Cooksley KDE Sysadmin
Re: How to prevent users from opening GitLab issues?
On Thu, Oct 10, 2019 at 3:26 PM Simon Redman wrote: > > You can add templates by following the directions here: > https://docs.gitlab.com/ee/user/project/description_templates.html#creating-issue-templates > > (Similar steps for pull requests, check out the kdeconnect-kde repo if you > want the templates I made) > > Unfortunately regular developers can't set the default. It has to be done by > an admin. At least I assume so because I have been looking for the repository > settings and I can't find them :) > > I wish we could make it required that a user pick *some* issue (and PR) > template. That way at least you could have a bug report "template" which says > "Don't report bugs here" Specifying a default issue template is an Enterprise Edition only feature i'm afraid. Please see https://docs.gitlab.com/ee/user/project/description_templates.html#setting-a-default-template-for-merge-requests-and-issues--starter for more information Regards, Ben > > On 10/9/19 11:30 AM, Alexander Saoutkin wrote: > > One should be able to add a issue template to issues (similar to how diffs > are prefilled in phab), which would make clear the purpose of issues. > > feverfew > > On Wed, 9 Oct 2019, 19:23 Nate Graham, wrote: >> >> GitLab Issues are intended to be the replacement for Phabricator tasks >> and are not intended to be used for user bug reports. We can't disable >> this feature or else we lose the project task tracking functionality >> entirely, which would be a major regression compared to Phabricator. >> >> Yes, it is very confusing to users, and yes, we are going to be dealing >> with users clogging it up with bug reports for the foreseeable future >> until we manage to rename Issues to "Project Tasks" or add a big banner >> directing users wanting to file bugs back to Bugzilla or something. >> >> Nate >> >> >> >> On 10/9/19 7:40 AM, Albert Vaca Cintora wrote: >> > As we transition to Gitlab, our users find two different ways of >> > reporting issues: in Bugzilla and in Gitlab. Since we don't have plans >> > to deprecate bugzilla, is there any way we can disable issues in Gitlab? >> > >> > Albert >> > >> >
Re: How to prevent users from opening GitLab issues?
On Thu, Oct 10, 2019 at 7:23 AM Nate Graham wrote: > > GitLab Issues are intended to be the replacement for Phabricator tasks > and are not intended to be used for user bug reports. We can't disable > this feature or else we lose the project task tracking functionality > entirely, which would be a major regression compared to Phabricator. > > Yes, it is very confusing to users, and yes, we are going to be dealing > with users clogging it up with bug reports for the foreseeable future > until we manage to rename Issues to "Project Tasks" or add a big banner > directing users wanting to file bugs back to Bugzilla or something. One possibility would be for us to enable Gitlab's Bugzilla integration, which would make Bugzilla more discoverable compared to Gitlab issues. https://docs.gitlab.com/ee/user/project/integrations/bugzilla.html We could also restrict Gitlab issues to those people who are members of the project. This would mean however that everyone who doesn't hold a Developer account (or given the Reporter role separately) is completely cut-off and unable to see our internal tasks. Changing the name of Issues would unfortunately require changes to the actual Gitlab source code in many places unfortunately (and would cause translation issues for those who had changed the language from English to something else) Introducing a custom banner would likely be less invasive, but would still require code changes i'm afraid. (Side note: in the past we have received complaints about Bugzilla making their email address public despite us having a warning in 100pt+ font that would happen as the first page in the registration process which is the first thing you saw - so people will miss a custom banner) > > Nate > Cheers, Ben > > > On 10/9/19 7:40 AM, Albert Vaca Cintora wrote: > > As we transition to Gitlab, our users find two different ways of > > reporting issues: in Bugzilla and in Gitlab. Since we don't have plans > > to deprecate bugzilla, is there any way we can disable issues in Gitlab? > > > > Albert > > >
Re: How to prevent users from opening GitLab issues?
On Fri, Oct 11, 2019 at 4:38 AM Nate Graham wrote: > > On 10/10/19 3:28 AM, Ben Cooksley wrote: > > Specifying a default issue template is an Enterprise Edition only > > feature i'm afraid. > > Please see > > https://docs.gitlab.com/ee/user/project/description_templates.html#setting-a-default-template-for-merge-requests-and-issues--starter > > for more information > > I must admit feeling rather frustrated that so may of the features which > make a GitHub/GitLab style of platform desirable are EE-only. Our > crippled CE version is missing these features as well as some of the > features that Phabricator currently has, such batching review comments, > formal approvals, strong relationships between tasks, and a web-based > notification system that doesn't send you ten thousand emails a day. > For the record, we currently have a request with them to move batching of review comments to CE. They've also already agreed to move approvals (not blocking) to CE. Once the batching is in I suspect your email problems will become less of an issue. (This isn't the first time i've mentioned the above two items either) Please note that an Enterprise Edition license would be available to us, if it weren't for Community Policy prohibiting the use of non-FOSS software. > Nate > Regards, Ben
Re: Get Hot New Stuff Dead or Alive?
On Mon, Oct 14, 2019 at 6:06 AM Allen Winter wrote: > > or maybe renamed? Hi Allen, > > How do I find out if there are any GHNS for KOrganizer. > The Import->GHNS menu in KOrganizer starts and I see the import dialog ok. > but that dialog is empty. > > I need to know if something is broken with the KOrganizer implementation > or if I should remove that feature. Or maybe just nothing to import, which > makes me think I should remove the feature. > > As a first starting point, i'd suggest asking what endpoint your implementation is reaching out to. Depending on when it was last updated, there are at least 3 different places KOrganizer could be trying to fetch from. Two of them are essentially static, unchangeable archives for legacy implementations now, while the other one is store.kde.org. Cheers, Ben
Re: Can we agree to change gitlab default behaviour from merge to fast-forward merge for all repos?
On Mon, Oct 14, 2019 at 1:42 PM Aleix Pol wrote: > > On Sun, Oct 13, 2019 at 10:57 PM Albert Astals Cid wrote: > > > > I find the merge behavior to be not what we've been doing in phabricator so > > given the idea is to maintain our workflows i'd appreciate if we can agree > > on continue doing the same. > > > > https://docs.gitlab.com/ee/user/project/merge_requests/fast_forward_merge.html > > +1 > > This is the enterprise version though, does this apply to the community? All of their docs are under the /ee/ path on docs.gitlab.com - for the most part they're fully applicable to CE though. Features that are EE only are marked with the corresponding EE versions (see https://docs.gitlab.com/ee/user/project/insights/ for instance) > > Adding sysadmin as CC, since they're the ones who will have to put it > to work in the end. We received a request for this a little while ago - please see https://phabricator.kde.org/T11190 While we did investigate this, as it turns out this is part of Gitlab that is non-customisable in terms of the default (regardless of your edition). What we will likely end up doing is changing all projects to that setting via the Gitlab API following the rollout of Gitlab to all projects and ensure we change that configuration for new projects as part of the project provisioning process. Please note that our current Gitlab setup should be considered to be an evaluation system, hence why we haven't looked into implementing this yet. In terms of this request though: 1) Work branches will eliminate most of the problems and pain you're experiencing here I suspect 2) Forcing rebases to land changes means that you will be unable to land changes that comprise more than 100 commits, as may be the case with larger bodies of work. In terms of why (2) happens, this is because our commit hooks, in order to prevent abuse/flooding of common spaces (such as #commits on Freenode, and the kde-comm...@kde.org mailing list) block attempts to push greater than 100 commits. From the perspective of the commit hooks, rebased commits are entirely new commits - and any commit coming from a work branch (once those are introduced) will also count as a new commit. Our hooks do operate on merges performed by Gitlab and will result in Gitlab returning 'Something went wrong' when you try to merge in changes greater than 100 commits as the hooks will block Gitlab from performing the (rebased) merge for you. In these cases, should we proceed down the path of not creating merge commits (and requiring rebases in the event of it being a non-fast forward) then projects will need to perform the merge manually in these instances, or ask Sysadmin to temporarily switch them back to the Gitlab default of creating merge commits in order to land that series of changes. > > Aleix Cheers, Ben
Re: Can we agree to change gitlab default behaviour from merge to fast-forward merge for all repos?
On Wed, 16 Oct 2019, 05:17 Johan Ouwerkerk, wrote: > On Tue, Oct 15, 2019 at 9:17 AM Frederik Schwarzer > wrote: > > Now I will fix my latest revision and merge to master. Still: 19 commits > > are not compiling anymore. > > > > Or am I missing something here? > > > > How would we deal with that? Is "short-lived branches" (as you stated > > below) enough to reduce the risk? > > > > To answer in reverse order: > > Yes. On the one hand short-lived branches reduce this risk > considerably: this scenario applies to breaking changes in master > which fundamentally alter the way the application works. It's like a > core library upgrade or, in this case, a major UX rewrite: something > fairly fundamental to the application changes in master. It is > unlikely that this should happen overnight without any kind of prior > review. > > Still, this can and does happen and will happen some day to you if you > contribute enough :) However this gets back at the git rebase bit. > > So yes, you rebase your feature on master as per normal, fix transient > merge conflicts and then what? Well, then you still have to compile & > test. At which point you notice breakage. How do you recover? As you > would: you begin the porting effort, either changes from master to > your new feature way of doing things (in case *you* are the one doing > the UX rewrite/major refactoring), or vice versa you apply the new > world order from master to your feature. > > What I like to do during this process is to avoid committing these > fixes just yet. I want to get a feel for the total diff, in particular > the total git diff --stat that I accumulate. Then I can identify on a > file-by-file basis using something like git log -3 path/to/file or so > what the likely commit is which should have been amended. Sometimes > you notice the diff for a file should be spread over multiple commits > according to your prior log, so what you do next is you use git add > like this: git add -p path/to/file. You only select the bits for which > you have identified a particular commit, you commit those added hunks > and here I like to leave a note in the first line of the commit to the > effect of "fixup " or "squash " or "". > > In this way you build up a bunch of commits which cover your fixes. > Next up, you turn to git rebase again, using e.g. git rebase -i > master. Now you can interactively fold the commits into the history as > "it ought to be" and this is where I use my notes to help me decide > how to proceed. Note you don't have to get everything just right, and > note that this rebasing itself may introduce transient merge conflicts > you need to fix: so if the diff stat was large it makes sense to split > this up into multiple git rebase -i runs just to give yourself a break > in between. Finally perhaps you rebase again to touch up a few commit > messages or something, and if this whole process took considerable > amount of time you want to verify that upstream master has not yet > moved on by that point. > > So in this more complex case you can adopt a correspondingly more > complex git workflow and use rebase to produce clean commits. Now, > sometimes you decide this is all too much work, and too much bother. > What you can do in such a scenario instead is to create a fresh new > branch from master and effectively re-create commits there. In those > cases git cherry-pick and git checkout -p -- path/to/file > come in handy > > Ultimately whether or not a scrupulously clean commit log is worth the > effort or whether you might decide to simplify things a little and > accept a few broken commits in between mostly depends on the needs of > your project and how many people work on it. > It was complexity of that degree that I was primarily concerned about when people started pushing for being able to force push and use a rebase workflow. For a first time contributor or anyone who isn't familiar with working with Git rebase, that sort of workflow is incredibly daunting, downright scary and is likely to push them in the direction of simply not contributing at all. As I'm sure we can all agree, that isn't something we want at all - we want it to be easy and straight forward to get involved for everyone. In cases such as the one you have above, I think a simple merge commit would be completely fine in that instance, especially given it is an uncommon event. > Regards, > > -Johan > Cheers, Ben >
Re: Can we agree to change gitlab default behaviour from merge to fast-forward merge for all repos?
On Thu, Oct 17, 2019 at 4:24 PM Roman Gilg wrote: > > On Tue, Oct 15, 2019 at 8:12 PM Ben Cooksley wrote: > > > > On Wed, 16 Oct 2019, 05:17 Johan Ouwerkerk, wrote: > >> [...] > > > > > > It was complexity of that degree that I was primarily concerned about when > > people started pushing for being able to force push and use a rebase > > workflow. > > > > For a first time contributor or anyone who isn't familiar with working with > > Git rebase, that sort of workflow is incredibly daunting, downright scary > > and is likely to push them in the direction of simply not contributing at > > all. > > > > As I'm sure we can all agree, that isn't something we want at all - we want > > it to be easy and straight forward to get involved for everyone. > > > > In cases such as the one you have above, I think a simple merge commit > > would be completely fine in that instance, especially given it is an > > uncommon event. > > It was like this on Phabricator until now so we should leave it like > that for the GitLab transition. > The rebase behaviour described by Johan is something you would need to do manually with feature branches, while the Gitlab behaviour in question in this thread is a fully automated one. That makes it quite different from what we had with Phabricator. (Side note: Phabricator doesn't carry commits, so anything landed is always squashed) > That being said the git merge workflow instead of rebase has certain > advantages and just because we did it until now with rebase does not > need to imply we have to continue to do so for all time. > > What I mean by that: if you set the default of GitLab merge behavior > to rebase, please do it in a way that allows individual projects to > override the default at some point in the future if seen fit. As noted previously, Gitlab does not allow setting a default for new projects - so what we will do is change the setting on all of our projects, and ensure we change it on all new projects going forward. I'm not sure whether we should support/permit projects to opt to a different workflow, as it is bound to cause someone to notice it is different and then file a Sysadmin ticket saying it's wrong because it is different to the policy applied to all other projects. This means that we (Sysadmin) then have to track who has opted to have different behaviour, and people are then bound to challenge why it is different when we close those tickets as invalid. Regards, Ben
Re: Looking for Subtitle Composer sponsor
On Fri, Nov 8, 2019 at 8:58 AM Nate Graham wrote: > > On 11/7/19 12:55 PM, Mladen Milinkovic wrote: > > Hello Nate, > > > > On 11/7/19 3:31 PM, Nate Graham wrote: > >> I would be happy to be your sponsor! > > Thank you! > > > >> Per https://community.kde.org/Incubator#Candidate, we need you to provide > >> the following information: > >> > >> - A list of the people regularly committing to the project > > That would be only me I believe. Right now I'm the only one with write > > permissions to repository. > > There are occasional pull requests, but nothing regular. Most pull requests > > are translations which since recently are > > pulled from crowdin.com. > > This should be the list of contributors since Nov 2011: > > https://github.com/maxrd2/SubtitleComposer/graphs/contributors > > > >> - A plan to be in compliance with the KDE manifesto > >> (https://manifesto.kde.org/index.html). In particular, you'll need > >> to be okay moving your source code repo and other infrastructure away from > >> GitHub and into KDE's GitLab instance at > >> https://invent.kde.org. > > Have already created account on https://invent.kde.org/users/milinkovic. > > Should I create a personal repository there for > > the project? Hi Nate, > > Sysadmin will get one set up for you. > > Sysadmin, can we have a repo to host > https://github.com/maxrd2/SubtitleComposer on KDE's infrastructure? We'll need a ticket submitted for this (as emails can get lost, but tickets cannot be lost). If you could submit one at https://go.kde.org/systickets that would be appreciated. > > Nate > Regards, Ben
Sysadmin Load Reduction: Code Related Services
Hi all, In the category of code related services, Sysadmin currently supports a wide-ranging number of services (which makes sense given the nature of the community). Some of these however may no longer be in use or will be duplicative of other services following the transition to Gitlab. In the category of duplicative, we have our two CGit instances at cgit.kde.org and packaging.neon.kde.org. Prior to Gitlab, these were justifiable as there was no other way of browsing scratch/clone repositories over the web. With the migration to Gitlab however all repositories will become browsable through it, meaning it no longer makes sense to offer two browsers that provide the exact same information (with Gitlab having greater capabilities). I'd therefore like to shut both of these down once Code Hosting has transitioned to Gitlab. In the category of no longer in use, we have the compatibility generator for the kde_projects.xml file. This was introduced when we shutdown Redmine/Chiliproject and migrated to Phabricator, as a way of keeping services that needed to discover a list of KDE Projects functional. As we've since migrated to using YAML files within the sysadmin/repo-metadata repository for both the CI System and kdesrc-build (and with LXR using kdesrc-build to do it's code checkouts) there shouldn't to my knowledge be anything still relying on this (aside from perhaps scripty). I'd therefore like to shut this generator down as well, along with the compaibility redirector running at projects.kde.org (given that it has been some time since we were using that site, and many projects have moved around in the virtual structure since then, making the redirects it is able to offer useless) Any comments regarding the above? Cheers, Ben
Sysadmin Load Reduction: Subversion Infrastructure
Hi all, When KDE committed to performing a migration to Git back in 2010, one of the things that was agreed at the time was that translators could remain on Subversion to avoid disrupting their workflows. This however has led to a certain amount of additional infrastructure which Sysadmin needs to continue to maintain. In recognition of the fact that with few exceptions, everything has now migrated from Subversion aside from Translations, i'd like to reduce the level of infrastructure supporting our Subversion repository to the bare minimum necessary. This would include the shutdown of WebSVN in particular, which when coupled with the shutdown of our two CGit instances would also allow for us to eliminate an entire virtual machine from our systems. On top of this, i'd also like to remove commit access to it for everyone but translators and those who need to work on the small number of websites remaining on Subversion and only provision this for people on an as-needed basis. In the next year or so i'd expect the remaining websites to complete their migrations to Git, after which only translators would receive access. We would also cease providing geographically distributed anonsvn service, with anonymous access only being provided by the master server going forward. Any comments? Thanks, Ben
General Infrastructure Maintainability: API and EBN
Hi all, Over the past number of years one of the tasks the Sysadmin team has worked on has been improving the overall maintainability of our systems, with a significant number of specialised cronjobs, exceptions and hidden linkages being eliminated. That is with one great exception: api.kde.org and ebn.kde.org. Both of these are suffering from an extreme amount of digital bitrot and special casing and in general are now in a condition where I cannot say for certain whether it would be possible to replicate the setup on a new system without us experiencing some degree of breakage (some of which we may not discover until weeks/months afterwards). In addition, the current setup relies on an old-fashioned overnight reprocessing of all repositories, which is inefficient and resource expensive. A more modern approach would have the various projects api documentation generated on a delayed cycle from relevant branches as part of something like a CI job (but not part of the actual CI workflow itself). For this one, i'm not certain on the best path forward at this stage, however the current state of affairs cannot continue. We have tried over the past few years to find people to work on a replacement for the tooling involved, but alas we've yet to have success here. Thoughts anyone? Regards, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 10, 2019 at 12:29 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > Hi all, > > > > In the category of code related services, Sysadmin currently supports > > a wide-ranging number of services (which makes sense given the nature > > of the community). Some of these however may no longer be in use or > > will be duplicative of other services following the transition to > > Gitlab. > > > > In the category of duplicative, we have our two CGit instances at > > cgit.kde.org and packaging.neon.kde.org. Prior to Gitlab, these were > > justifiable as there was no other way of browsing scratch/clone > > repositories over the web. > > > > With the migration to Gitlab however all repositories will become > > browsable through it, meaning it no longer makes sense to offer two > > browsers that provide the exact same information (with Gitlab having > > greater capabilities). I'd therefore like to shut both of these down > > once Code Hosting has transitioned to Gitlab. > > Luckily we tried to use commits.kde.org as generic redirect. Will the generic > redirect commits.kde.org be updated to point to the proper repository? It may > be complicated if the new structure contains the namespaces for each > repository, but we need to keep it working. The commits.kde.org redirector is intended to provide permanent links, so yes it will be updated to redirect people to Gitlab instead once the switch to Gitlab is completed. > > > > > In the category of no longer in use, we have the compatibility > > generator for the kde_projects.xml file. This was introduced when we > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way of > > keeping services that needed to discover a list of KDE Projects > > functional. > > > > As we've since migrated to using YAML files within the > > sysadmin/repo-metadata repository for both the CI System and > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > checkouts) there shouldn't to my knowledge be anything still relying > > on this (aside from perhaps scripty).> > > I'd therefore like to shut this generator down as well, along with the > > compaibility redirector running at projects.kde.org (given that it has > > been some time since we were using that site, and many projects have > > moved around in the virtual structure since then, making the redirects > > it is able to offer useless) > > Two points: > - scripty still uses the XML file and we may need some time to migrate. I feared this may have been the case. What sort of timeline are we looking at in terms of switching scripty over? > - we may have a few links around which still points to projects.kde.org (for > example, the "Consistency" goal listed a few of them), so we may need to go > through all of them before doing that. > Do we have measures that shows how much projects.kde.org is hit by services > which are not scripty? I've reviewed the logs for the past 24 hours and it would appear the majority of the legitimate hits now are for the RSS/ATOM feeds that Chiliproject used to offer and which have no corresponding replacement (amazingly, these are still configured in people's feed readers). After excluding bots and spiders, and people running automated tools against old URL sets, very little in the way of hits remained. The remainder are a very small proportion of hits from places such as the Archlinux wiki, which we can likely request to be updated (referencing Bluedevil in particular it would seem). > > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 12:20 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > Hi all, > > > > When KDE committed to performing a migration to Git back in 2010, one > > of the things that was agreed at the time was that translators could > > remain on Subversion to avoid disrupting their workflows. > > > > This however has led to a certain amount of additional infrastructure > > which Sysadmin needs to continue to maintain. > > > > In recognition of the fact that with few exceptions, everything has > > now migrated from Subversion aside from Translations, i'd like to > > reduce the level of infrastructure supporting our Subversion > > repository to the bare minimum necessary. > > > > This would include the shutdown of WebSVN in particular, which when > > coupled with the shutdown of our two CGit instances would also allow > > for us to eliminate an entire virtual machine from our systems. > > As websvn has proven to be useful, would it help the sysadmin (pending > agreement with aacid and the other people with root access on rosetta) to move > websvn to rosetta and under localization team's responsibility? Unfortunately Rosetta does not have the necessary free disk space, as the Subversion repository is approximately 80GB these days in size, and operating WebSVN requires a full copy of the Subversion repository locally in order to operate. > > Ciao > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev wrote: > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > This would include the shutdown of WebSVN in particular, which when > > coupled with the shutdown of our two CGit instances would also allow > > for us to eliminate an entire virtual machine from our systems. > > Will there be any web interface for SVN after shutdown of WebSVN? > > Can we assume https://phabricator.kde.org/source/svn/ remains > available during the next 10 years? > Phabricator's browser will be retired as part of the shutdown of Phabricator, which will take place once Gitlab has assumed responsibility for code hosting and review, and the tasks have been migrated from Phabricator. Should WebSVN be shutdown as well, then there would be no web interface to our SVN repository. > > I'm not sure what use case Luigi has in mind, but among Russian l10n > team we often use WebSVN to refer to specific SVN revisions, for > example: https://websvn.kde.org/?view=revision&revision=1555342 > > -- > Alexander Potashev Regards, Ben
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 4:28 AM Elvis Angelaccio wrote: > > > > On 09/11/19 01:33, Ben Cooksley wrote: > > Hi all, > > Hi Ben Hi Elvis, > > > > > Over the past number of years one of the tasks the Sysadmin team has > > worked on has been improving the overall maintainability of our > > systems, with a significant number of specialised cronjobs, exceptions > > and hidden linkages being eliminated. > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > Both of these are suffering from an extreme amount of digital bitrot > > and special casing and in general are now in a condition where I > > cannot say for certain whether it would be possible to replicate the > > setup on a new system without us experiencing some degree of breakage > > (some of which we may not discover until weeks/months afterwards). > > > > In addition, the current setup relies on an old-fashioned overnight > > reprocessing of all repositories, which is inefficient and resource > > expensive. A more modern approach would have the various projects api > > documentation generated on a delayed cycle from relevant branches as > > part of something like a CI job (but not part of the actual CI > > workflow itself). > > > > For this one, i'm not certain on the best path forward at this stage, > > however the current state of affairs cannot continue. We have tried > > over the past few years to find people to work on a replacement for > > the tooling involved, but alas we've yet to have success here. > > > > Thoughts anyone? > > I'd say api.kde.org is too important to let it go. The EBN is less > important but still useful, I still see people pushing EBN-based fixes > once in a while. > > Have we ever tried to use a GSoC project to modernize either of them? If > not, would it make sense to try next year? > My only concern there would be whether the project is large enough in scope to justify the amount of time commitment a GSoC project normally spans. >From my understanding of the problem in question it quite probably does not meet those requirements. > > > > Regards, > > Ben > > Cheers, > Elvis > Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > wrote: > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > This would include the shutdown of WebSVN in particular, which when > > > > coupled with the shutdown of our two CGit instances would also allow > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > available during the next 10 years? > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > Phabricator, which will take place once Gitlab has assumed > > responsibility for code hosting and review, and the tasks have been > > migrated from Phabricator. > > > > Should WebSVN be shutdown as well, then there would be no web > > interface to our SVN repository. > > That's not acceptable. Mind explaining why? Bear in mind that there is a cost both in terms of infrastructure, and people time to maintain a service such as WebSVN. This includes having to maintain a mirror of the repository on the machine that runs WebSVN, along with the associated infrastructure for allowing that mirroring to happen on the master server. > > Cheers, > Albert Regards, Ben > > > > > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > > team we often use WebSVN to refer to specific SVN revisions, for > > > example: https://websvn.kde.org/?view=revision&revision=1555342 > > > > > > -- > > > Alexander Potashev > > > > Regards, > > Ben > > > > > >
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 10, 2019 at 8:43 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 18:47:57 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 12:29 AM Luigi Toscano > > wrote: > > > > > > Ben Cooksley ha scritto: > > > > Hi all, > > > > > > > > In the category of code related services, Sysadmin currently supports > > > > a wide-ranging number of services (which makes sense given the nature > > > > of the community). Some of these however may no longer be in use or > > > > will be duplicative of other services following the transition to > > > > Gitlab. > > > > > > > > In the category of duplicative, we have our two CGit instances at > > > > cgit.kde.org and packaging.neon.kde.org. Prior to Gitlab, these were > > > > justifiable as there was no other way of browsing scratch/clone > > > > repositories over the web. > > > > > > > > With the migration to Gitlab however all repositories will become > > > > browsable through it, meaning it no longer makes sense to offer two > > > > browsers that provide the exact same information (with Gitlab having > > > > greater capabilities). I'd therefore like to shut both of these down > > > > once Code Hosting has transitioned to Gitlab. > > > > > > Luckily we tried to use commits.kde.org as generic redirect. Will the > > > generic > > > redirect commits.kde.org be updated to point to the proper repository? It > > > may > > > be complicated if the new structure contains the namespaces for each > > > repository, but we need to keep it working. > > > > The commits.kde.org redirector is intended to provide permanent links, > > so yes it will be updated to redirect people to Gitlab instead once > > the switch to Gitlab is completed. > > > > > > > > > > > > > In the category of no longer in use, we have the compatibility > > > > generator for the kde_projects.xml file. This was introduced when we > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way of > > > > keeping services that needed to discover a list of KDE Projects > > > > functional. > > > > > > > > As we've since migrated to using YAML files within the > > > > sysadmin/repo-metadata repository for both the CI System and > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > checkouts) there shouldn't to my knowledge be anything still relying > > > > on this (aside from perhaps scripty).> > > > > I'd therefore like to shut this generator down as well, along with the > > > > compaibility redirector running at projects.kde.org (given that it has > > > > been some time since we were using that site, and many projects have > > > > moved around in the virtual structure since then, making the redirects > > > > it is able to offer useless) > > > > > > Two points: > > > - scripty still uses the XML file and we may need some time to migrate. > > > > I feared this may have been the case. What sort of timeline are we > > looking at in terms of switching scripty over? > > Over to what? To using the YAML files within sysadmin/repo-metadata, which is what the legacy kde_projects.xml file is generated from currently. (Originally the kde_projects.xml file was generated from the Redmine/Chiliproject database - those YAML files just replaced it) > > Cheers, > Albert > > Thanks, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 9:32 AM Albert Vaca Cintora wrote: > > On Sat, Nov 9, 2019 at 6:53 PM Ben Cooksley wrote: > > > > Unfortunately Rosetta does not have the necessary free disk space, as > > the Subversion repository is approximately 80GB these days in size, > > and operating WebSVN requires a full copy of the Subversion repository > > locally in order to operate. > > Is this also the case if you run WebSVN next to the SVN server itself > (ie: on the same machine)? If a single repo copy can be shared by > WebSVN and the actual SVN server, that would reduce the storage needs. Running WebSVN on the same machine is not safe for us to do, because certain requests cause WebSVN to generate a substantial amount of system load, and have caused out of memory events on systems running it in the past. This poses an unacceptable risk to the canonical copy of our repositories, as well as to Gitlab (which will be hosted on the same machine as the master Subversion repository) > > Albert Regards, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 10, 2019 at 11:37 AM Alexander Potashev wrote: > > сб, 9 нояб. 2019 г. в 02:51, Ben Cooksley : > > In the category of no longer in use, we have the compatibility > > generator for the kde_projects.xml file. This was introduced when we > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way of > > keeping services that needed to discover a list of KDE Projects > > functional. > > > > As we've since migrated to using YAML files within the > > sysadmin/repo-metadata repository for both the CI System and > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > checkouts) there shouldn't to my knowledge be anything still relying > > on this (aside from perhaps scripty). > > > > I'd therefore like to shut this generator down as well, along with the > > compaibility redirector running at projects.kde.org (given that it has > > been some time since we were using that site, and many projects have > > moved around in the virtual structure since then, making the redirects > > it is able to offer useless) > > Hi Ben, Hi Alexander, > > I am developing a new version of the "opensrc" plugin for Lokalize [1] > and it currently depends on kde_projects.xml. Of course I can add new > code to scan the Git repo instead of just fetching kde_projects.xml, > however it would be more complicated. > > > Is there any good reason for killing kde_projects.xml? Would that > really reduce the workload on Sysadmin team? > I can't suspect any "moving parts" in the generator that may require > maintenance. > This is mostly a case of another (small) thing that needs to be kept running, and fixed should it break. All of the reductions i've proposed will have a quite small effect, however cumulatively they will have a much larger effect. Given that it is duplicative of what we have in the YAML files (which are substantially easier to maintain) it makes more sense to standardise on the original source of the information and remove the legacy compatibility XML file. > > [1] https://cgit.kde.org/scratch/aspotashev/lokalize-opensrc-py.git/ > > -- > Alexander Potashev Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 10, 2019 at 11:22 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 21:52:13 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 8:43 AM Albert Astals Cid wrote: > > > > > > El dissabte, 9 de novembre de 2019, a les 18:47:57 CET, Ben Cooksley va > > > escriure: > > > > On Sun, Nov 10, 2019 at 12:29 AM Luigi Toscano > > > > wrote: > > > > > > > > > > Ben Cooksley ha scritto: > > > > > > Hi all, > > > > > > > > > > > > In the category of code related services, Sysadmin currently > > > > > > supports > > > > > > a wide-ranging number of services (which makes sense given the > > > > > > nature > > > > > > of the community). Some of these however may no longer be in use or > > > > > > will be duplicative of other services following the transition to > > > > > > Gitlab. > > > > > > > > > > > > In the category of duplicative, we have our two CGit instances at > > > > > > cgit.kde.org and packaging.neon.kde.org. Prior to Gitlab, these were > > > > > > justifiable as there was no other way of browsing scratch/clone > > > > > > repositories over the web. > > > > > > > > > > > > With the migration to Gitlab however all repositories will become > > > > > > browsable through it, meaning it no longer makes sense to offer two > > > > > > browsers that provide the exact same information (with Gitlab having > > > > > > greater capabilities). I'd therefore like to shut both of these down > > > > > > once Code Hosting has transitioned to Gitlab. > > > > > > > > > > Luckily we tried to use commits.kde.org as generic redirect. Will the > > > > > generic > > > > > redirect commits.kde.org be updated to point to the proper > > > > > repository? It may > > > > > be complicated if the new structure contains the namespaces for each > > > > > repository, but we need to keep it working. > > > > > > > > The commits.kde.org redirector is intended to provide permanent links, > > > > so yes it will be updated to redirect people to Gitlab instead once > > > > the switch to Gitlab is completed. > > > > > > > > > > > > > > > > > > > > > In the category of no longer in use, we have the compatibility > > > > > > generator for the kde_projects.xml file. This was introduced when we > > > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way > > > > > > of > > > > > > keeping services that needed to discover a list of KDE Projects > > > > > > functional. > > > > > > > > > > > > As we've since migrated to using YAML files within the > > > > > > sysadmin/repo-metadata repository for both the CI System and > > > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > > > checkouts) there shouldn't to my knowledge be anything still relying > > > > > > on this (aside from perhaps scripty).> > > > > > > I'd therefore like to shut this generator down as well, along with > > > > > > the > > > > > > compaibility redirector running at projects.kde.org (given that it > > > > > > has > > > > > > been some time since we were using that site, and many projects have > > > > > > moved around in the virtual structure since then, making the > > > > > > redirects > > > > > > it is able to offer useless) > > > > > > > > > > Two points: > > > > > - scripty still uses the XML file and we may need some time to > > > > > migrate. > > > > > > > > I feared this may have been the case. What sort of timeline are we > > > > looking at in terms of switching scripty over? > > > > > > Over to what? > > > > To using the YAML files within sysadmin/repo-metadata, which is what > > the legacy kde_projects.xml file is generated from currently. > > Well, someone has to change the code that parses the xml to code that gets > stuff from git and parses yaml, it's not hard, but has to be done. *nod* I do recall that at one point in our conversations that scripty only used the kde_projects.xml file as a check against it's own internal data. Has that since been changed? > > > (Originally the kde_projects.xml file was generated from the > > Redmine/Chiliproject database - those YAML files just replaced it) > > Yes i know :) > > Cheers, > Albert Cheers, Ben > > > > > > > > > Cheers, > > > Albert > > > > > > > > > > Thanks, > > Ben > > > > > >
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 10, 2019 at 1:24 PM Albert Astals Cid wrote: > > El diumenge, 10 de novembre de 2019, a les 1:04:04 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 11:22 AM Albert Astals Cid wrote: > > > > > > El dissabte, 9 de novembre de 2019, a les 21:52:13 CET, Ben Cooksley va > > > escriure: > > > > On Sun, Nov 10, 2019 at 8:43 AM Albert Astals Cid wrote: > > > > > > > > > > El dissabte, 9 de novembre de 2019, a les 18:47:57 CET, Ben Cooksley > > > > > va escriure: > > > > > > On Sun, Nov 10, 2019 at 12:29 AM Luigi Toscano > > > > > > wrote: > > > > > > > > > > > > > > Ben Cooksley ha scritto: > > > > > > > > Hi all, > > > > > > > > > > > > > > > > In the category of code related services, Sysadmin currently > > > > > > > > supports > > > > > > > > a wide-ranging number of services (which makes sense given the > > > > > > > > nature > > > > > > > > of the community). Some of these however may no longer be in > > > > > > > > use or > > > > > > > > will be duplicative of other services following the transition > > > > > > > > to > > > > > > > > Gitlab. > > > > > > > > > > > > > > > > In the category of duplicative, we have our two CGit instances > > > > > > > > at > > > > > > > > cgit.kde.org and packaging.neon.kde.org. Prior to Gitlab, these > > > > > > > > were > > > > > > > > justifiable as there was no other way of browsing scratch/clone > > > > > > > > repositories over the web. > > > > > > > > > > > > > > > > With the migration to Gitlab however all repositories will > > > > > > > > become > > > > > > > > browsable through it, meaning it no longer makes sense to offer > > > > > > > > two > > > > > > > > browsers that provide the exact same information (with Gitlab > > > > > > > > having > > > > > > > > greater capabilities). I'd therefore like to shut both of these > > > > > > > > down > > > > > > > > once Code Hosting has transitioned to Gitlab. > > > > > > > > > > > > > > Luckily we tried to use commits.kde.org as generic redirect. Will > > > > > > > the generic > > > > > > > redirect commits.kde.org be updated to point to the proper > > > > > > > repository? It may > > > > > > > be complicated if the new structure contains the namespaces for > > > > > > > each > > > > > > > repository, but we need to keep it working. > > > > > > > > > > > > The commits.kde.org redirector is intended to provide permanent > > > > > > links, > > > > > > so yes it will be updated to redirect people to Gitlab instead once > > > > > > the switch to Gitlab is completed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the category of no longer in use, we have the compatibility > > > > > > > > generator for the kde_projects.xml file. This was introduced > > > > > > > > when we > > > > > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a > > > > > > > > way of > > > > > > > > keeping services that needed to discover a list of KDE Projects > > > > > > > > functional. > > > > > > > > > > > > > > > > As we've since migrated to using YAML files within the > > > > > > > > sysadmin/repo-metadata repository for both the CI System and > > > > > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > > > > > checkouts) there shouldn't to my knowledge be anything still > > > > > > > > relying > > > > > > > > on this (aside from perhaps scripty).> > > > > > > > > I'd therefore like to shut this generator down as well,
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 11:47 AM Friedrich W. H. Kossebau wrote: > > Am Samstag, 9. November 2019, 21:22:16 CET schrieb Ben Cooksley: > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > escriure: > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > wrote: > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > > available during the next 10 years? > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > > Phabricator, which will take place once Gitlab has assumed > > > > responsibility for code hosting and review, and the tasks have been > > > > migrated from Phabricator. > > > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > > interface to our SVN repository. > > > > > > That's not acceptable. > > > > Mind explaining why? > > FWIW, everytime I had to deal with translations as developer (like checking > pot files as well as .po files contents) I found having the web interface and > its browsing feature very valuable to quickly find what I was looking for, > over having to locally mess around with svn commands and juggling between > commandline & file viewers. Including url bookmarks for quick access to > browsing certain sets of files. > > Incidents which I remember right now included: > * finding out whether extraction scripts were working as intended > * comparing translations seen by users over what they should see > > Are there any other KDE clients of the svn repos still around, besides > translation system? The only other clients I am aware of are a small number of websites, which can be found at /trunk/www/ in SVN. The list of ones we serve can be found at https://invent.kde.org/sysadmin/binary-factory-tooling/blob/master/staticweb/standard-jobs.yaml#L69 > Perhaps the "full clone" needed for WebSVN could be reduced to the translation > subtrees, would that improve situation to a degree if possible? (well, you > surely thought of this yourself, just in case) Unfortunately WebSVN needs a full copy of the underlying repository to be able to work. > > For me as developer contributor to projects in KDE spheres, losing the web > browsing interface for raw translation files would be a regression in > developer experience. > > Cheers > Friedrich > > Regards, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > > > > > > El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > > > escriure: > > > > On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > > > > wrote: > > > > > > > > > > сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > > > > > > This would include the shutdown of WebSVN in particular, which when > > > > > > coupled with the shutdown of our two CGit instances would also allow > > > > > > for us to eliminate an entire virtual machine from our systems. > > > > > > > > > > Will there be any web interface for SVN after shutdown of WebSVN? > > > > > > > > > > Can we assume https://phabricator.kde.org/source/svn/ remains > > > > > available during the next 10 years? > > > > > > > > > > > > > Phabricator's browser will be retired as part of the shutdown of > > > > Phabricator, which will take place once Gitlab has assumed > > > > responsibility for code hosting and review, and the tasks have been > > > > migrated from Phabricator. > > > > > > > > Should WebSVN be shutdown as well, then there would be no web > > > > interface to our SVN repository. > > > > > > That's not acceptable. > > > > Mind explaining why? > > Because we use it in l10n.kde.org to link to po files. Mind detailing what those links are used for? > > > Bear in mind that there is a cost both in terms of infrastructure, and > > people time to maintain a service such as WebSVN. > > We have money, we don't have to shut down things we use because there is a > cost. I wasn't referring to monetary cost there, I was referring to the flow on effects (such as having to maintain the necessary components on the master server to allow for the Subversion repository to be mirrored). Note also the "people time" component there. > > Cheers, > Albert Thanks, Ben > > > This includes having to maintain a mirror of the repository on the > > machine that runs WebSVN, along with the associated infrastructure for > > allowing that mirroring to happen on the master server. > > > > > > > > Cheers, > > > Albert > > > > Regards, > > Ben > > > > > > > > > > > > > > > > > > > I'm not sure what use case Luigi has in mind, but among Russian l10n > > > > > team we often use WebSVN to refer to specific SVN revisions, for > > > > > example: https://websvn.kde.org/?view=revision&revision=1555342 > > > > > > > > > > -- > > > > > Alexander Potashev > > > > > > > > Regards, > > > > Ben > > > > > > > > > > > > > > > > > > > > > >
Re: Sysadmin Load Reduction: Subversion Infrastructure
On Mon, Nov 11, 2019 at 3:58 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > On Sun, Nov 10, 2019 at 11:19 AM Albert Astals Cid wrote: > >> > >> El dissabte, 9 de novembre de 2019, a les 21:22:16 CET, Ben Cooksley va > >> escriure: > >>> On Sun, Nov 10, 2019 at 8:37 AM Albert Astals Cid wrote: > >>>> > >>>> El dissabte, 9 de novembre de 2019, a les 19:27:07 CET, Ben Cooksley va > >>>> escriure: > >>>>> On Sun, Nov 10, 2019 at 7:18 AM Alexander Potashev > >>>>> wrote: > >>>>>> > >>>>>> сб, 9 нояб. 2019 г. в 03:20, Ben Cooksley : > >>>>>>> This would include the shutdown of WebSVN in particular, which when > >>>>>>> coupled with the shutdown of our two CGit instances would also allow > >>>>>>> for us to eliminate an entire virtual machine from our systems. > >>>>>> > >>>>>> Will there be any web interface for SVN after shutdown of WebSVN? > >>>>>> > >>>>>> Can we assume https://phabricator.kde.org/source/svn/ remains > >>>>>> available during the next 10 years? > >>>>>> > >>>>> > >>>>> Phabricator's browser will be retired as part of the shutdown of > >>>>> Phabricator, which will take place once Gitlab has assumed > >>>>> responsibility for code hosting and review, and the tasks have been > >>>>> migrated from Phabricator. > >>>>> > >>>>> Should WebSVN be shutdown as well, then there would be no web > >>>>> interface to our SVN repository. > >>>> > >>>> That's not acceptable. > >>> > >>> Mind explaining why? > >> > >> Because we use it in l10n.kde.org to link to po files. > > > > Mind detailing what those links are used for? > > > >> > >>> Bear in mind that there is a cost both in terms of infrastructure, and > >>> people time to maintain a service such as WebSVN. > >> > >> We have money, we don't have to shut down things we use because there is a > >> cost. > > > > I wasn't referring to monetary cost there, I was referring to the flow > > on effects (such as having to maintain the necessary components on the > > master server to allow for the Subversion repository to be mirrored). > > > > Note also the "people time" component there. > > Sure, but please see my previous questions: > - can we extend the space of rosetta? It already has a partial checkout, and > 100 GB of free space (which can be kept down We would need to ask the system hosters to provide this, as Rosetta is a donated system (if memory serves, Rosetta is currently IOPS constrainted, so using it to host WebSVN may over-burden the system) > - if that's not enough, can we simply setup a machine which periodically sync > from the svn repository? You are probably going to tell me that it does not > work without server support, but from what I'm reading about svnsync, I don't > think it should overload the server if executed every 30 minutes. > Are we sure that we still need something on the master server? Let's try it > first. I'm afraid you cannot use svnsync with KDE's Subversion repository, that utility hasn't been able to handle it for many, many years now (it crashes if memory serves - [ade] hit this and documented it on his blog back around the time it broke which was before the switch to Git got underway) We have custom tooling which uses rsync instead to mirror the repository. Custom tooling is necessary because plain rsync cannot be used to reliably mirror the repository - because it has 1.5 million commits in it, which means over 3 million inodes on disk. Each of these would have to be checked by plain usage of rsync, which takes a substantial amount of time even on NVMe SSD storage - during which time the local mirror of the repository may be inconsistent (rendering it unusable), and which also generates a matching disk load on the master server, reducing it's performance. The RSync endpoint for the mirrors to contact is the component on the master server that has to be maintained. > - in case it's not clear, I'm volunteering to maintain that system. > > Ciao > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Subversion Infrastructure
Hi all, This has been quite the long thread for one that was started only around 2 days ago, so thought I would summarise things: On the topic of removal of the Subversion mirrors, there have been no objections, so we will proceed with this. With regards to removal of access for everyone but translators, there have been two objections noted to this. One concerned compliance with the Manifesto, whilst the other was concerned with creating obstacles. Given that restricting access to the websites is still compliant with the Manifesto despite the lack of a clear clause permitting this I don't believe that not granting access by default to Subversion should be an issue, so would like to proceed with this as well. That leaves the last point, which has generated the most contention, the removal of WebSVN (websvn.kde.org). Based on the responses, it would appear that translation workflows are currently heavily embedded with and tightly dependent on this service. This would indicate that we would need to keep this service around for now. With just a few websites left on Subversion though, I would like to know if the translation teams have a plan for how they will continue to do things in the long term, and whether this includes a migration away from Subversion (as otherwise we are maintaining indefinitely a Subversion repository, customised hooks, mirroring infrastructure, WebSVN, and support for the email notifications from those customised hooks within other systems such as our IRC Bots and Bugzilla) Regards, Ben
Re: Looking for Subtitle Composer sponsor
On Thu, Nov 14, 2019 at 8:53 PM Mladen Milinkovic wrote: > > On 11/13/19 7:14 PM, Nate Graham wrote: > > On 11/8/19 12:24 PM, Nate Graham wrote: > >> On 11/8/19 2:42 AM, Ben Cooksley wrote: > >>> We'll need a ticket submitted for this (as emails can get lost, but > >>> tickets cannot be lost). > >>> If you could submit one at https://go.kde.org/systickets that would be > >>> appreciated. > >> > >> Done, I've filed https://phabricator.kde.org/T11998 > >> > >> Nate > >> > > > > Looks like the repo is open and you've got your commit rights, so you can > > start migrating to use that repo now, Mladen! > > > > Nate > > That's great, thank you. How/where can I access the repo? > I don't have permissions to see https://phabricator.kde.org/T11998 That is a Sysadmin ticket, which is why you wouldn't have been able to access it as Nate filed that for you. The repository can be found at https://invent.kde.org/kde/subtitlecomposer > > Mladen Cheers, Ben
Re: Looking for Subtitle Composer sponsor
On Thu, Nov 14, 2019 at 9:31 PM Mladen Milinkovic wrote: > > On 11/14/19 8:54 AM, Ben Cooksley wrote: > > On Thu, Nov 14, 2019 at 8:53 PM Mladen Milinkovic > > wrote: > >> > >> On 11/13/19 7:14 PM, Nate Graham wrote: > >>> > >>> Looks like the repo is open and you've got your commit rights, so you can > >>> start migrating to use that repo now, Mladen! > >>> > >>> Nate > >> > >> That's great, thank you. How/where can I access the repo? > >> I don't have permissions to see https://phabricator.kde.org/T11998 > > > > That is a Sysadmin ticket, which is why you wouldn't have been able to > > access it as Nate filed that for you. > > The repository can be found at https://invent.kde.org/kde/subtitlecomposer > > > >> > >> Mladen > > > > Cheers, > > Ben > > > > Thank you. It doesn't seem I have commit rights to it tho. > I get this when trying to push: > > > GitLab: You are not allowed to push code to this project. > fatal: Could not read from remote repository. > > Please make sure you have the correct access rights > and the repository exists. > > Also am not listed in > https://invent.kde.org/kde/subtitlecomposer/-/settings/members Sorry about that, that occurs when someone receives their developer account after logging into Gitlab for the first time. I've now corrected your group membership. (This is something we intend on fixing as part of the final deployment) > > Mladen Thanks, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sat, Nov 16, 2019 at 3:27 AM Alexander Potashev wrote: > > пт, 15 нояб. 2019 г. в 15:46, Harald Sitter : > > > > On Sat, Nov 9, 2019 at 11:37 PM Alexander Potashev > > wrote: > > > > > > сб, 9 нояб. 2019 г. в 02:51, Ben Cooksley : > > > > In the category of no longer in use, we have the compatibility > > > > generator for the kde_projects.xml file. This was introduced when we > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way of > > > > keeping services that needed to discover a list of KDE Projects > > > > functional. > > > > > > > > As we've since migrated to using YAML files within the > > > > sysadmin/repo-metadata repository for both the CI System and > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > checkouts) there shouldn't to my knowledge be anything still relying > > > > on this (aside from perhaps scripty). > > > > > > > > I'd therefore like to shut this generator down as well, along with the > > > > compaibility redirector running at projects.kde.org (given that it has > > > > been some time since we were using that site, and many projects have > > > > moved around in the virtual structure since then, making the redirects > > > > it is able to offer useless) > > > > > > Hi Ben, > > > > > > I am developing a new version of the "opensrc" plugin for Lokalize [1] > > > and it currently depends on kde_projects.xml. Of course I can add new > > > code to scan the Git repo instead of just fetching kde_projects.xml, > > > however it would be more complicated. > > > > https://projects.kde.org/api/doc/ specifically deals with this problem > > by abstracting the repo away behind a micro service. > > This looks like another view of the data available in > kde_projects.xml, however the API is very limited. For example I can't > query repo descriptions using this API. Thus not helpful. > > However if we were going to kill kde_projects.xml, are you sure > projects.kde.org/api/ would be still available and not shut down as > well? The API that Harald mentions is based off the YAML files, and is not reliant on the legacy kde_projects.xml file. > > -- > Alexander Potashev Cheers, Ben
Disabling of 'testkioarchive' in kio-extras
Hi all, Currently we have an issue where the test 'testkioarchive' within kio-extras causes kdeinit5 to be relaunched, as can be seen in the CI run log below. https://build.kde.org/job/Applications/job/kio-extras/job/kf5-qt5%20SUSEQt5.12/40/console This is behaviour which is not permitted within the CI system, as the newly spawned 'kdeinit5' processes are treated as being launched by that test by CTest, meaning it will wait for eternity for them to exit (which as resident daemon processes, they will not do without manual intervention). This in turn blocks the job from completing, and blocks CI resources in the process. I'd therefore like to propose disabling this test across all platforms as it's behaviour is incompatible with the CI system. It is the sole and exclusive responsibility of the CI tooling to ensure that kdeinit5 is started with the appropriate environment, and this should not be tampered with by tests under any circumstances. Regards, Ben
Disabling akonadi test runner infrastructure across all repositories
Hi all, For some time now the Akonadi Test Runner infrastructure has had issues on the CI system where it will fail to properly shutdown the akonadiserver (and associated subprocesses) that it starts up for tests. This leads to jobs on the CI system becoming indefinitely stuck as CTest will wait indefinitely for subprocesses spawned by tests to exit. This in turn requires manual cleanup. Tonight jobs for kdepim-runtime and akonadi itself both required manual assistance to complete. Other PIM repositories have in the past required similar assistance. Given that the issue is not limited to a single test or repository, i'd therefore like to propose no-oping the entire CMake macro responsible for the akonadi test runner framework and disabling all tests that make use of it globally across all repositories until this issue can be rectified. Regards, Ben
Disabling of 'kdav-davitemfetchjob' test in KDav
Hi everyone, Currently we have an issue where the test 'kdav-davitemfetchjob' within kdav causes kdeinit5 to be relaunched, as can be seen in the CI run log below. https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.12/38/console This is behaviour which is not permitted within the CI system, as the newly spawned 'kdeinit5' processes are treated as being launched by that test by CTest, meaning it will wait for eternity for them to exit (which as resident daemon processes, they will not do without manual intervention). This in turn blocks the job from completing, and blocks CI resources in the process. I'd therefore like to propose disabling this test across all platforms as it's behaviour is incompatible with the CI system. Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sat, Nov 16, 2019 at 3:58 PM Alexander Potashev wrote: > > пт, 15 нояб. 2019 г. в 22:05, Ben Cooksley : > > > > On Sat, Nov 16, 2019 at 3:27 AM Alexander Potashev > > wrote: > > > > > > пт, 15 нояб. 2019 г. в 15:46, Harald Sitter : > > > > > > > > On Sat, Nov 9, 2019 at 11:37 PM Alexander Potashev > > > > wrote: > > > > > > > > > > сб, 9 нояб. 2019 г. в 02:51, Ben Cooksley : > > > > > > In the category of no longer in use, we have the compatibility > > > > > > generator for the kde_projects.xml file. This was introduced when we > > > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a way > > > > > > of > > > > > > keeping services that needed to discover a list of KDE Projects > > > > > > functional. > > > > > > > > > > > > As we've since migrated to using YAML files within the > > > > > > sysadmin/repo-metadata repository for both the CI System and > > > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > > > checkouts) there shouldn't to my knowledge be anything still relying > > > > > > on this (aside from perhaps scripty). > > > > > > > > > > > > I'd therefore like to shut this generator down as well, along with > > > > > > the > > > > > > compaibility redirector running at projects.kde.org (given that it > > > > > > has > > > > > > been some time since we were using that site, and many projects have > > > > > > moved around in the virtual structure since then, making the > > > > > > redirects > > > > > > it is able to offer useless) > > > > > > > > > > Hi Ben, > > > > > > > > > > I am developing a new version of the "opensrc" plugin for Lokalize [1] > > > > > and it currently depends on kde_projects.xml. Of course I can add new > > > > > code to scan the Git repo instead of just fetching kde_projects.xml, > > > > > however it would be more complicated. > > > > > > > > https://projects.kde.org/api/doc/ specifically deals with this problem > > > > by abstracting the repo away behind a micro service. > > > > > > This looks like another view of the data available in > > > kde_projects.xml, however the API is very limited. For example I can't > > > query repo descriptions using this API. Thus not helpful. > > > > > > However if we were going to kill kde_projects.xml, are you sure > > > projects.kde.org/api/ would be still available and not shut down as > > > well? > > > > The API that Harald mentions is based off the YAML files, and is not > > reliant on the legacy kde_projects.xml file. > > I can implement kde_projects.xml or a modernized version of it as part > of projects.kde.org/api/. Does this sound like a good idea? > > We don't need to support the exact same XML format, so I would prefer > a JSON listing all projects with all their properties, at something > like GET https://projects.kde.org/api/v1/export or may be return all > project properties in GET https://projects.kde.org/api/v1/projects I'll leave it to Harald to comment on whether he'd be happy having additional capabilities in the Projects Microservice API he maintains. (Harald, I assume it's only requirement is a copy of the sysadmin/repo-metadata repository locally, and it doesn't require the actual Git repositories?) We really should avoid continuing to keep legacy endpoints alive though (as it is just shifting the maintenance effort of porting away from it further down the road), especially for something that is going to end up on end user systems. On that note, should you be using QNetworkAccessManager, please ensure you forcibly and explicitly enable handling of redirects. > > -- > Alexander Potashev Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Sat, Nov 16, 2019 at 10:39 PM Albert Astals Cid wrote: > > El dissabte, 16 de novembre de 2019, a les 10:14:15 CET, Ben Cooksley va > escriure: > > On Sat, Nov 16, 2019 at 3:58 PM Alexander Potashev > > wrote: > > > > > > пт, 15 нояб. 2019 г. в 22:05, Ben Cooksley : > > > > > > > > On Sat, Nov 16, 2019 at 3:27 AM Alexander Potashev > > > > wrote: > > > > > > > > > > пт, 15 нояб. 2019 г. в 15:46, Harald Sitter : > > > > > > > > > > > > On Sat, Nov 9, 2019 at 11:37 PM Alexander Potashev > > > > > > wrote: > > > > > > > > > > > > > > сб, 9 нояб. 2019 г. в 02:51, Ben Cooksley : > > > > > > > > In the category of no longer in use, we have the compatibility > > > > > > > > generator for the kde_projects.xml file. This was introduced > > > > > > > > when we > > > > > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, as a > > > > > > > > way of > > > > > > > > keeping services that needed to discover a list of KDE Projects > > > > > > > > functional. > > > > > > > > > > > > > > > > As we've since migrated to using YAML files within the > > > > > > > > sysadmin/repo-metadata repository for both the CI System and > > > > > > > > kdesrc-build (and with LXR using kdesrc-build to do it's code > > > > > > > > checkouts) there shouldn't to my knowledge be anything still > > > > > > > > relying > > > > > > > > on this (aside from perhaps scripty). > > > > > > > > > > > > > > > > I'd therefore like to shut this generator down as well, along > > > > > > > > with the > > > > > > > > compaibility redirector running at projects.kde.org (given that > > > > > > > > it has > > > > > > > > been some time since we were using that site, and many projects > > > > > > > > have > > > > > > > > moved around in the virtual structure since then, making the > > > > > > > > redirects > > > > > > > > it is able to offer useless) > > > > > > > > > > > > > > Hi Ben, > > > > > > > > > > > > > > I am developing a new version of the "opensrc" plugin for > > > > > > > Lokalize [1] > > > > > > > and it currently depends on kde_projects.xml. Of course I can add > > > > > > > new > > > > > > > code to scan the Git repo instead of just fetching > > > > > > > kde_projects.xml, > > > > > > > however it would be more complicated. > > > > > > > > > > > > https://projects.kde.org/api/doc/ specifically deals with this > > > > > > problem > > > > > > by abstracting the repo away behind a micro service. > > > > > > > > > > This looks like another view of the data available in > > > > > kde_projects.xml, however the API is very limited. For example I can't > > > > > query repo descriptions using this API. Thus not helpful. > > > > > > > > > > However if we were going to kill kde_projects.xml, are you sure > > > > > projects.kde.org/api/ would be still available and not shut down as > > > > > well? > > > > > > > > The API that Harald mentions is based off the YAML files, and is not > > > > reliant on the legacy kde_projects.xml file. > > > > > > I can implement kde_projects.xml or a modernized version of it as part > > > of projects.kde.org/api/. Does this sound like a good idea? > > > > > > We don't need to support the exact same XML format, so I would prefer > > > a JSON listing all projects with all their properties, at something > > > like GET https://projects.kde.org/api/v1/export or may be return all > > > project properties in GET https://projects.kde.org/api/v1/projects > > > > I'll leave it to Harald to comment on whether he'd be happy having > > additional capabilities in the Projects Microservice API he maintains. > > (Harald, I assume it's only requirement is a copy of the > > sysadmin/repo-metadata repository locally, and it doesn't require the > > actual Git repositories?) > > > > We really should avoid continuing to keep legacy endpoints alive > > though (as it is just shifting the maintenance effort of porting away > > from it further down the road), especially for something that is going > > to end up on end user systems. > > Wait, are you saying we shouldn't be using that endpoint to solve our > dependency on kde_projects.xml on scripty either? > > I just asked Adrián to have a look at it, since it seemed easier than having > to worry about downloading and keeping an up to date copy of another repo. I was referring to the "implement kde_projects.xml" part of his email there. The modern components of the projects.kde.org/api/ endpoint are of course fine. > > Cheers, > Albert Cheers, Ben > > > > > On that note, should you be using QNetworkAccessManager, please ensure > > you forcibly and explicitly enable handling of redirects. > > > > > > > > -- > > > Alexander Potashev > > > > Cheers, > > Ben > > > > > >
Re: Looking for Subtitle Composer sponsor
On Sun, Nov 17, 2019 at 12:12 AM Mladen Milinkovic wrote: > > On 11/14/19 9:47 AM, Ben Cooksley wrote: > > On Thu, Nov 14, 2019 at 9:31 PM Mladen Milinkovic > > wrote: > >> Thank you. It doesn't seem I have commit rights to it tho. > >> I get this when trying to push: > >> > >>> GitLab: You are not allowed to push code to this project. > >> fatal: Could not read from remote repository. > >> > >> Please make sure you have the correct access rights > >> and the repository exists. > >> > >> Also am not listed in > >> https://invent.kde.org/kde/subtitlecomposer/-/settings/members > > > > Sorry about that, that occurs when someone receives their developer > > account after logging into Gitlab for the first time. > > I've now corrected your group membership. > > > > (This is something we intend on fixing as part of the final deployment) > > > > Thank you. > > I was trying to setup release builds using CI. > Mind elaborating on the specifics of what you're trying to do here? At the moment the CI capabilities of Gitlab are for testing purposes only and shouldn't be relied on. We do intend to transition to Gitlab CI but that will happen after all the repositories transition to Gitlab. Should you wish to have CI services, this can be done using build.kde.org instead. > I don't have permissions on the new repo to access settings (eg. so I can't > add CI/CD env variables). Should I file a > ticket somewhere when I want to change something in repo settings or will I > get higher permissions at some point? You will need to file Sysadmin tickets (at https://go.kde.org/systickets) to have repository settings changed i'm afraid. > > I have noticed that GitLab API for creating tags is not working. It doesn't > even work from web interface. Both just give > blank errror message and HTTP/1.1 400 Bad Request. Is this a bug or > intentional? > API URL: > https://invent.kde.org/api/v4/projects/731/repository/tags?ref=master&tag_name=test > WEB URL: https://invent.kde.org/milinkovic/subtitlecomposer/-/tags/new The API should work in theory. You've been providing all the necessary parameters as specified at https://docs.gitlab.com/ee/api/tags.html#create-a-new-tag I assume? > > Mladen Cheers, Ben
Re: Disabling akonadi test runner infrastructure across all repositories
On Sun, Nov 17, 2019 at 7:48 AM Simon Redman wrote: > > My only thought about this is, is it possible to disable the tests in a > way which leaves a warning so that at least it isn't silently "passing"? If they are disabled in the way I am suggesting, they wouldn't even be registered with CTest. Cheers, Ben > > On 11/15/19 10:02 PM, Ben Cooksley wrote: > > Hi all, > > > > For some time now the Akonadi Test Runner infrastructure has had > > issues on the CI system where it will fail to properly shutdown the > > akonadiserver (and associated subprocesses) that it starts up for > > tests. > > > > This leads to jobs on the CI system becoming indefinitely stuck as > > CTest will wait indefinitely for subprocesses spawned by tests to > > exit. This in turn requires manual cleanup. > > > > Tonight jobs for kdepim-runtime and akonadi itself both required > > manual assistance to complete. Other PIM repositories have in the past > > required similar assistance. > > > > Given that the issue is not limited to a single test or repository, > > i'd therefore like to propose no-oping the entire CMake macro > > responsible for the akonadi test runner framework and disabling all > > tests that make use of it globally across all repositories until this > > issue can be rectified. > > > > Regards, > > Ben >
Re: Sysadmin Load Reduction: Code Related Services
On Sun, Nov 17, 2019 at 5:19 AM Carl Schwan wrote: > > Hi all, Hi Carl, > > Can the gitlab api be of useful in the future? > > e.g https://invent.kde.org/api/v4/groups/7 While for many purposes Gitlab's API wil be perfectly fine, it doesn't provide us with the i18n branch information which some users will require. > > Cheers, > Carl > Cheers, Ben > ‐‐‐ Original Message ‐‐‐ > On Saturday, November 16, 2019 9:51 AM, Ben Cooksley > wrote: > > > On Sat, Nov 16, 2019 at 10:39 PM Albert Astals Cid aa...@kde.org wrote: > > > > > > El dissabte, 16 de novembre de 2019, a les 10:14:15 CET, Ben Cooksley va > > > escriure: > > > > > > > > On Sat, Nov 16, 2019 at 3:58 PM Alexander Potashev aspotas...@gmail.com > > > > wrote: > > > > > > > > > > пт, 15 нояб. 2019 г. в 22:05, Ben Cooksley bcooks...@kde.org: > > > > > > > > > > > > On Sat, Nov 16, 2019 at 3:27 AM Alexander Potashev > > > > > > aspotas...@gmail.com wrote: > > > > > > > > > > > > > > пт, 15 нояб. 2019 г. в 15:46, Harald Sitter sit...@kde.org: > > > > > > > > > > > > > > > > On Sat, Nov 9, 2019 at 11:37 PM Alexander Potashev > > > > > > > > aspotas...@gmail.com wrote: > > > > > > > > > > > > > > > > > > сб, 9 нояб. 2019 г. в 02:51, Ben Cooksley bcooks...@kde.org: > > > > > > > > > > > > > > > > > > > > In the category of no longer in use, we have the > > > > > > > > > > compatibility > > > > > > > > > > generator for the kde_projects.xml file. This was > > > > > > > > > > introduced when we > > > > > > > > > > shutdown Redmine/Chiliproject and migrated to Phabricator, > > > > > > > > > > as a way of > > > > > > > > > > keeping services that needed to discover a list of KDE > > > > > > > > > > Projects > > > > > > > > > > functional. > > > > > > > > > > As we've since migrated to using YAML files within the > > > > > > > > > > sysadmin/repo-metadata repository for both the CI System and > > > > > > > > > > kdesrc-build (and with LXR using kdesrc-build to do it's > > > > > > > > > > code > > > > > > > > > > checkouts) there shouldn't to my knowledge be anything > > > > > > > > > > still relying > > > > > > > > > > on this (aside from perhaps scripty). > > > > > > > > > > I'd therefore like to shut this generator down as well, > > > > > > > > > > along with the > > > > > > > > > > compaibility redirector running at projects.kde.org (given > > > > > > > > > > that it has > > > > > > > > > > been some time since we were using that site, and many > > > > > > > > > > projects have > > > > > > > > > > moved around in the virtual structure since then, making > > > > > > > > > > the redirects > > > > > > > > > > it is able to offer useless) > > > > > > > > > > > > > > > > > > > Hi Ben, > > > > > > > > > I am developing a new version of the "opensrc" plugin for > > > > > > > > > Lokalize [1] > > > > > > > > > and it currently depends on kde_projects.xml. Of course I can > > > > > > > > > add new > > > > > > > > > code to scan the Git repo instead of just fetching > > > > > > > > > kde_projects.xml, > > > > > > > > > however it would be more complicated. > > > > > > > > > > > > > > > > > https://projects.kde.org/api/doc/ specifically deals with this > > > > > > > > problem > > > > > > > > by abstracting the repo away behind a micro service. > > > > > > > > > > > > > > > This looks like another view of the data available in > > > > > > > kde_projects.xml, however the API i
Re: Disabling akonadi test runner infrastructure across all repositories
On Mon, Nov 18, 2019 at 10:18 AM Volker Krause wrote: > > On Sunday, 17 November 2019 21:20:11 CET Albert Astals Cid wrote: > > El dissabte, 16 de novembre de 2019, a les 7:02:04 CET, Ben Cooksley va > escriure: > > > Hi all, > > > > > > For some time now the Akonadi Test Runner infrastructure has had > > > issues on the CI system where it will fail to properly shutdown the > > > akonadiserver (and associated subprocesses) that it starts up for > > > tests. > > > > > > This leads to jobs on the CI system becoming indefinitely stuck as > > > CTest will wait indefinitely for subprocesses spawned by tests to > > > exit. This in turn requires manual cleanup. > > > > > > Tonight jobs for kdepim-runtime and akonadi itself both required > > > manual assistance to complete. Other PIM repositories have in the past > > > required similar assistance. > > > > > > Given that the issue is not limited to a single test or repository, > > > i'd therefore like to propose no-oping the entire CMake macro > > > responsible for the akonadi test runner framework and disabling all > > > tests that make use of it globally across all repositories until this > > > issue can be rectified. > > > > I think it would be really sad to lose all these tests, what's the timeframe > > we're speaking here to try to fix them? > > see replies to the other threads on CI test hangs, I've pushed a possible fix > (taken from KIO) to the affected repos. Now waiting to see if that's effective > and whether all places were found. The test hang in the case of Akonadi and associated repositories is unrelated to kdeinit5/klauncher/etc issues, as the problem in this instance is the akonadiserver instance launched by the test framework not being shutdown and cleaned up prior to the test and it's associated wrappers exiting. I'm therefore not sure if the same fix will apply here... > > Regards, > Volker Cheers, Ben
Re: Disabling akonadi test runner infrastructure across all repositories
On Mon, Nov 18, 2019 at 9:45 PM Volker Krause wrote: > > On Monday, 18 November 2019 02:55:19 CET Ben Cooksley wrote: > > On Mon, Nov 18, 2019 at 10:18 AM Volker Krause wrote: > > > On Sunday, 17 November 2019 21:20:11 CET Albert Astals Cid wrote: > > > > El dissabte, 16 de novembre de 2019, a les 7:02:04 CET, Ben Cooksley va > > > > > > escriure: > > > > > Hi all, > > > > > > > > > > For some time now the Akonadi Test Runner infrastructure has had > > > > > issues on the CI system where it will fail to properly shutdown the > > > > > akonadiserver (and associated subprocesses) that it starts up for > > > > > tests. > > > > > > > > > > This leads to jobs on the CI system becoming indefinitely stuck as > > > > > CTest will wait indefinitely for subprocesses spawned by tests to > > > > > exit. This in turn requires manual cleanup. > > > > > > > > > > Tonight jobs for kdepim-runtime and akonadi itself both required > > > > > manual assistance to complete. Other PIM repositories have in the past > > > > > required similar assistance. > > > > > > > > > > Given that the issue is not limited to a single test or repository, > > > > > i'd therefore like to propose no-oping the entire CMake macro > > > > > responsible for the akonadi test runner framework and disabling all > > > > > tests that make use of it globally across all repositories until this > > > > > issue can be rectified. > > > > > > > > I think it would be really sad to lose all these tests, what's the > > > > timeframe we're speaking here to try to fix them? > > > > > > see replies to the other threads on CI test hangs, I've pushed a possible > > > fix (taken from KIO) to the affected repos. Now waiting to see if that's > > > effective and whether all places were found. > > > > The test hang in the case of Akonadi and associated repositories is > > unrelated to kdeinit5/klauncher/etc issues, as the problem in this > > instance is the akonadiserver instance launched by the test framework > > not being shutdown and cleaned up prior to the test and it's > > associated wrappers exiting. I'm therefore not sure if the same fix > > will apply here... > > Right, that issue might also still be there. The hang that triggered this > discussion in kdepim-runtime however seemed to be due to KIO operations in the > POP3 and EWS tests (similar to what we saw in KDAV and KIO Extras). It's also > possible I misread the logs though :) > > In case the problem persists after all, my original question on how to detect > we are running on the CI becomes relevant again. Is there an environment > variable that would be a sufficiently good indicator for this? I'd really like > to avoid removing useful tests that work perfectly fine locally. Detecting you are running in the CI system is a bit difficult i'm afraid, as the CI system takes deliberate measures to appear to be a normal system. There are a number of variables set by Jenkins itself you can probably detect though: https://build.kde.org/job/Applications/job/kdepim-runtime/job/kf5-qt5%20SUSEQt5.12/70/consoleText Note that when we transition to Gitlab that these variables would no longer be set as it has it's own set of variables. > > Regards, > Volker Cheers, Ben
Changes to CI System and Binary Factory
Hi all, Over the past few days we have completed a series of changes to how the Binary Factory and CI system operate, with a view to improving scalability and security of the systems going forward. As a consequence of these changes, we should now be able to scale Flatpak builds across to multiple builders, more easily provision Android signing keys, provide Android builds in our F-Droid testing repository more rapidly and have the possibility of providing Snap builds in the future. Additionally, it will allow us to increase the number of FreeBSD and Windows builders (from the existing 2 of each to 3 of each), and should make the provisioning process for builders easier in the future. These changes however may have caused some breakages, particularly with regards to Android packages, due to the nature of the changes which have been introduced. Should anyone have any issues, please let us know. Cheers, Ben Cooksley KDE Sysadmin
Re: Sysadmin Load Reduction: Code Related Services
On Tue, Nov 19, 2019 at 1:34 AM Harald Sitter wrote: > > On Sat, Nov 16, 2019 at 9:40 PM Ben Cooksley wrote: > > > > On Sun, Nov 17, 2019 at 5:19 AM Carl Schwan wrote: > > > > > > Hi all, > > > > Hi Carl, > > > > > > > > Can the gitlab api be of useful in the future? > > > > > > e.g https://invent.kde.org/api/v4/groups/7 > > > > While for many purposes Gitlab's API wil be perfectly fine, it doesn't > > provide us with the i18n branch information which some users will > > require. > > Something to perhaps consider at this point is to revise the > repo-metadata format in general and offload data to gitlab? Once we have transitioned repository hosting and code review to Gitlab, restructuring how repo-metadata works was one of the items I wanted to look into (if anything because i'd like to automate updates to repo-metadata so we don't have to create them on Gitlab and then register them in repo-metadata as a second separate step) > > Notably I'd kick the following yaml properties in favor of their > literal replacements in gitlab: description, icon, members, name. On a Nothing uses members, so once the XML is gone we can get rid of that completely. Likewise, nothing uses icon either to my knowledge. Description and Name should definitely both come from Gitlab. > related note I guess there'd be need to figure out how to map from a > "kde project" to a gitlab project. Currently the paths between > repo-metadata and invent do not line up. The 'repopath' key in repo-metadata will serve this purpose (it might be missing the kde/ prefix though depending on how the implementation ends up working out) This fits into an overall discussion i'd like to have around playground/release service/extragear in the Git world though which will be brought up separately. > > Additionally we may able able to remove: > > - hasrepo (repopath==null implies this) This can go yes. > - repoactive (totally not sure what this communicates) This is meant to indicate whether a given entry/repository should be processed by CI and scripty. We have a number of unmaintained and other infrastructural repositories which both of those should ignore. > > And lastly, fold the i18n data into the yaml. Which I think means we > could drop the excessive use of directories and just have one file per I've no objection to this. > project, e.g. projects/kde/workspace/sddm-kcm.yaml or perhaps even > more ideally projects/sddm-kcm.yaml (because the current dir structure > duplicates information encoded in repopath; so I would think either we > should drop the property or flatten projects/). We should mirror the repopath as in the Gitlab world it will be possible for us to have two repositories that have the same name, just in different parts of the tree. > > HS Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Tue, Nov 19, 2019 at 6:20 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > On Tue, Nov 19, 2019 at 1:34 AM Harald Sitter wrote: > >> > >> On Sat, Nov 16, 2019 at 9:40 PM Ben Cooksley wrote: > >>> > >>> On Sun, Nov 17, 2019 at 5:19 AM Carl Schwan wrote: > >>>> > >>>> Hi all, > >>> > >>> Hi Carl, > >>> > >>>> > >>>> Can the gitlab api be of useful in the future? > >>>> > >>>> e.g https://invent.kde.org/api/v4/groups/7 > >>> > >>> While for many purposes Gitlab's API wil be perfectly fine, it doesn't > >>> provide us with the i18n branch information which some users will > >>> require. > >> > >> Something to perhaps consider at this point is to revise the > >> repo-metadata format in general and offload data to gitlab? > > > > Once we have transitioned repository hosting and code review to > > Gitlab, restructuring how repo-metadata works was one of the items I > > wanted to look into (if anything because i'd like to automate updates > > to repo-metadata so we don't have to create them on Gitlab and then > > register them in repo-metadata as a second separate step) > > Uhm, does gitlab allow to define custom properties? That may solve the > problem. (but probably it doesn't, or it would have already proposed.) It doesn't allow us to define custom properties i'm afraid. > > >> project, e.g. projects/kde/workspace/sddm-kcm.yaml or perhaps even > >> more ideally projects/sddm-kcm.yaml (because the current dir structure > >> duplicates information encoded in repopath; so I would think either we > >> should drop the property or flatten projects/). > > > > We should mirror the repopath as in the Gitlab world it will be > > possible for us to have two repositories that have the same name, just > > in different parts of the tree. > > Can we keep the rule of having unique repository names, even if we could use > them? One of the plans to reduce the moves around for translations is to > flatten the structure without namespaces. Allowing duplicates would make this > impossible and introduce more complication (and it may complicate our life as > well if a duplicated repository ever moves to plasma or to frameworks). It'll be hard to tell whether a name is unique or not when creating a repository unfortunately. For the most part I do not expect collisions to occur though and we certainly won't aim to create duplicate names. Note that I do not expect repositories to ever really move beyond the module they're created in (as a multimedia application will always be a multimedia application for instance). The release component (playground, extragear, release service) will not be encoded in the path which should greatly minimize the number of moves. Plasma and Frameworks will be two exceptions there, but that is because they're both modules and release units. > > -- > Luigi Cheers, Ben
Re: Disabling akonadi test runner infrastructure across all repositories
On Mon, Nov 18, 2019 at 11:44 PM Daniel Vrátil wrote: > > On Saturday, 16 November 2019 07:02:04 CET Ben Cooksley wrote: > > Hi all, > > hi Ben, Hi Dan, > > > > > For some time now the Akonadi Test Runner infrastructure has had > > issues on the CI system where it will fail to properly shutdown the > > akonadiserver (and associated subprocesses) that it starts up for > > tests. > > > > This leads to jobs on the CI system becoming indefinitely stuck as > > CTest will wait indefinitely for subprocesses spawned by tests to > > exit. This in turn requires manual cleanup. > > > > Tonight jobs for kdepim-runtime and akonadi itself both required > > manual assistance to complete. Other PIM repositories have in the past > > required similar assistance. > > > > Given that the issue is not limited to a single test or repository, > > i'd therefore like to propose no-oping the entire CMake macro > > responsible for the akonadi test runner framework and disabling all > > tests that make use of it globally across all repositories until this > > issue can be rectified. > > Disabling the tests that launch Akonadi server internally should be possible > by setting these CMake options in affected repos: > > -DAKONADI_RUN_MYSQL_ISOLATED_TESTS=FALSE > -DAKONADI_RUN_SQLITE_ISOLATED_TESTS=FALSE > -DAKONADI_RUN_PGSQL_ISOLATED_TESTS=FALSE > > It's possible that there are some repos where this options are not honored, so > should it look like setting those options had no effect, please let us know > and > we'll make sure the repo is ported to the correct CMake macro so that the > options have an effect. Wouldn't this have the same effect as disabling all the tests for ECM did though? (ie. become something we forget about, think all the tests are passing and then later find out a large number have regressed/broken) > > Regards, > Dan > Cheers, Ben > > > > Regards, > > Ben > > > -- > Daniel Vrátil > www.dvratil.cz | dvra...@kde.org > IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde) > > GPG Key: 0x4D69557AECB13683 > Fingerprint: 0ABD FA55 A4E6 BEA9 9A83 EA97 4D69 557A ECB1 3683
Re: Sysadmin Load Reduction: Code Related Services
On Tue, Nov 19, 2019 at 7:27 AM Albert Astals Cid wrote: > > El dilluns, 18 de novembre de 2019, a les 13:34:19 CET, Harald Sitter va > escriure: > > And lastly, fold the i18n data into the yaml. > > So to know which is the stable i18n branch of okular i have to checkout it's > master branch, read a file and then change to whatever branch it is said > there to be the stable branch? > > Feels a bit weird that the master branch has that kind of knowledge to be > honest. I think Harald was referring to the 'metadata.yaml' files in sysadmin/repo-metadata in this case. > > Cheers, > Albert > > Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Tue, Nov 19, 2019 at 6:53 AM Luigi Toscano wrote: > > Ben Cooksley ha scritto: > > On Tue, Nov 19, 2019 at 6:20 AM Luigi Toscano > > wrote: > >> > >> Ben Cooksley ha scritto: > >>> On Tue, Nov 19, 2019 at 1:34 AM Harald Sitter wrote: > >>>> > >>>> On Sat, Nov 16, 2019 at 9:40 PM Ben Cooksley wrote: > >>>>> > >>>>> On Sun, Nov 17, 2019 at 5:19 AM Carl Schwan wrote: > >>>>>> > >>>>>> Hi all, > >>>>> > >>>>> Hi Carl, > >>>>> > >>>>>> > >>>>>> Can the gitlab api be of useful in the future? > >>>>>> > >>>>>> e.g https://invent.kde.org/api/v4/groups/7 > >>>>> > >>>>> While for many purposes Gitlab's API wil be perfectly fine, it doesn't > >>>>> provide us with the i18n branch information which some users will > >>>>> require. > >>>> > >>>> Something to perhaps consider at this point is to revise the > >>>> repo-metadata format in general and offload data to gitlab? > >>> > >>> Once we have transitioned repository hosting and code review to > >>> Gitlab, restructuring how repo-metadata works was one of the items I > >>> wanted to look into (if anything because i'd like to automate updates > >>> to repo-metadata so we don't have to create them on Gitlab and then > >>> register them in repo-metadata as a second separate step) > >> > >> Uhm, does gitlab allow to define custom properties? That may solve the > >> problem. (but probably it doesn't, or it would have already proposed.) > > > > It doesn't allow us to define custom properties i'm afraid. > > Uhm, there is something, but it requires administrative access and I'm not > sure whether it's available in the community edition: > > https://docs.gitlab.com/ee/api/custom_attributes.html >From what I can tell this is available in the Community Edition. It is however though only available via the API (and isn't shown anywhere in the Gitlab UI), and of course can only be changed by an administrator (ie. Sysadmin) > > >>>> project, e.g. projects/kde/workspace/sddm-kcm.yaml or perhaps even > >>>> more ideally projects/sddm-kcm.yaml (because the current dir structure > >>>> duplicates information encoded in repopath; so I would think either we > >>>> should drop the property or flatten projects/). > >>> > >>> We should mirror the repopath as in the Gitlab world it will be > >>> possible for us to have two repositories that have the same name, just > >>> in different parts of the tree. > >> > >> Can we keep the rule of having unique repository names, even if we could > >> use > >> them? One of the plans to reduce the moves around for translations is to > >> flatten the structure without namespaces. Allowing duplicates would make > >> this > >> impossible and introduce more complication (and it may complicate our life > >> as > >> well if a duplicated repository ever moves to plasma or to frameworks). > > > > It'll be hard to tell whether a name is unique or not when creating a > > repository unfortunately. > > For the most part I do not expect collisions to occur though and we > > certainly won't aim to create duplicate names. > > I understand that, but still it means that we can't flatten the translation > repositories removing the namespaces. > Won't you still have issues though when repositories are renamed though? Moves into Plasma and Frameworks from already reviewed repositories are fairly uncommon from my recollection (about as common as repository renames) > Unless you use the future flattened translation repository as a way to see the > existing names. In fact, it should be possible to periodically (every two > days?) generate a list of all repository names; that should allow to check for > duplicates. Maybe this could reduce the minimum amount of uncertainty when > creating new repositories. > > -- > Luigi Cheers, Ben
Re: Sysadmin Load Reduction: Code Related Services
On Wed, Nov 20, 2019 at 1:48 AM Harald Sitter wrote: > > On Mon, Nov 18, 2019 at 6:29 PM Ben Cooksley wrote: > > It'll be hard to tell whether a name is unique or not when creating a > > repository unfortunately. > > For the most part I do not expect collisions to occur though and we > > certainly won't aim to create duplicate names. > > Wouldn't creating a repo also include creating the repo-metadata file > for it? Through that it should be easy to assert uniqueness via a > server-side hook on repo-metadata. Not necessarily. It will definitely be possible for a repository to exist on Gitlab but not in the repo-metadata repository. Any sync to repo-metadata would be a step taken after the repository is created on Gitlab. Ideally we would have an automated script sync the contents of repo-metadata after they're changed on Gitlab (ie. Gitlab is the authoritative source here) Having the repository come into existence after the repo-metadata entry can cause issues, although they appear fairly rarely as there is generally a delay between Sysadmin creating the repository and Developers starting to use a repository - hence why i'd like for the repository to appear first. > > In point of fact, if we fully flatten out the files I expect uniquness > would be implicit. > > e.g. > > repo-metadata/projects/blinken.yaml (repopath: kde/blinken) > repo-metadata/projects/kinfocenter.yaml (repopath: kde/kinfocenter > repo-metadata/projects/www-kde-org.yaml (repopath: sysadmin/www-kde-org) Website repositories, much like Sysadmin repositories, will continue to live in their own namespaces and therefore should not be flattened (if they're even included in repo-metadata, as they won't be under kde/) Please note that the repository paths for Blinken and KInfoCenter will be kde/edu/blinken and kde/plasma/kinfocenter respectively, as a degree of grouping is required on Gitlab for ease of visibility over merge requests and tasks (modules rarely change, and Gitlab handles redirecting old names to new names for us in any event even for Git clients) > > HS Cheers, Ben
Gitlab Update
Hi all, This afternoon we completed the move of Gitlab from it's initial server (which was for the evaluation) to the production server that will hopefully serve us well for many years to come. As part of this we also upgraded it from Gitlab 12.3 to 12.5. Should anyone notice any problems, please let us know. As part of the move we also enabled support for Incoming Email on our Gitlab instance. This means that tasks and merge requests can now be created by sending emails to certain addresses (which are available from the Gitlab web interface), and discussions on Gitlab can now be replied to by email as well. See https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html#create-new-merge-requests-by-email and https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#new-issue-via-email for more information on these features. Please note that the addresses are specific to you and allow anyone with them to perform actions as if they were you, so it is important that they are not disclosed to anyone else. Thanks, Ben Cooksley KDE Sysadmin
Blacklisting of PIM from the CI system
Hi all, This morning I went to look into provisioning a new Windows Builder node for the CI system, but hit a roadblock created by PIM currently failing to build from source. As some background to this, we use Craft to provide various dependencies of our projects that aren't provided by Windows itself (such as Qt, gettext, poppler and so forth). From time to time this is updated to provide newer versions of software, but unless it is necessary for a project, the CI nodes themselves often aren't updated. We use the same approach for both Linux and FreeBSD, except those dependencies are provided by the distribution for those two. This means that when it comes time to provision a new node, all of the nodes need to be updated. As these changes essentially always break compatibility in some form or another this makes it necessary for us to rebuild all the KDE software as well. It is therefore not possible to proceed with any of the above while something is failing to build. Which is where the problem with PIM comes in - because it currently has many repositories failing to build from source on all platforms those builds are enabled for (including Linux and FreeBSD). Given that the PIM project mailing list is emailed by the CI system, and that the changes in one case were pushed over 2 days ago, there is no excuse for this series of build failures. In addition to all of the above, this round of updates was to lay the ground work for adding additional dependencies which are necessary for the builds of Digikam and SubtitleComposer to commence on the CI system for Windows. These failures by PIM have therefore indirectly harmed other KDE projects. As this is not the first time that PIM has caused issues in this manner, I would therefore like to proceed with blacklisting PIM from the CI system. This would include prohibiting other projects from depending on any part of PIM, so those projects that have a required dependency on PIM would also have their builds removed by this. Whilst I would prefer another solution to this, given that it is a recurring issue that makes maintenance of the CI system substantially harder, I see the removal of PIM from the equation as the only reasonable path forward. Should the PIM developers wish to avoid this consequence for their actions, they will need to provide an action plan as to how this will be avoided in the future. Fixing the current set of failures will not prevent this blacklisting action from being carried out - as a recurring issue it needs a permanent solution. Regards, Ben Cooksley KDE Sysadmin
Re: Blacklisting of PIM from the CI system
On Sun, Dec 1, 2019 at 10:17 AM Volker Krause wrote: > > sigh... > > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: > [...] > > Which is where the problem with PIM comes in - because it currently > > has many repositories failing to build from source on all platforms > > those builds are enabled for (including Linux and FreeBSD). > > looking at this right now, at least three errors seem transient (ie. a manual > rebuild "fixed" them), one I can't explain yet is this one: > > https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/job/ > kf5-qt5%20SUSEQt5.12/39/console > > This is about using new API in KF 5.65, but the use is guarded by a > corresponding version check #ifdef. Which KF5 is the CI building against, > latest release, latest master, or maybe something in between (the latter would > explain this)? > The CI system essentially uses the latest master of Frameworks - with the slight catch of it the build being up to a week out of date. The Frameworks build is updated by the "Dependency Build" jobs that run for each Product/Branch/Platform combination once a week. The last time the Dependency Build job ran for the Applications kf5-qt5 SUSEQt5.12 combination was about 2 days ago. https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20kf5-qt5%20SUSEQt5.12/40/console Unfortunately this run failed due to breakage in 'pimcommon'. Before it failed, it did get the chance to build Sonnet so that is up to date as of the date of that build. > I'll look through the rest as time permits. Thanks, that would be appreciated. > > > Given that the PIM project mailing list is emailed by the CI system, > > and that the changes in one case were pushed over 2 days ago, there is > > no excuse for this series of build failures. > > I try to actively look at the CI state every 24h, however that sometimes fails > when I'm on the road or otherwise busy. I do see the email notifications, but Thanks for doing so. > given the high number of transient failures (usually due to unfortunately > timed dependency bumps), they are of limited use for me for raising an alarm > in cases of actual breakage. *nod* With regards to the dependency bumps, the best way to avoid these is to first bump version numbers, and then increase the dependency requirements in the projects in a second round of commits, pushed separately, once the version bump builds have finished. This is from my understanding how David does it for Frameworks and it works flawlessly. This solution has been proposed in the past, but was rejected by the PIM team who insisted that it was the CI system that should instead sequence the builds correctly (something which is unscalable to implement, and from my understanding impossible with Gitlab CI) > > > In addition to all of the above, this round of updates was to lay the > > ground work for adding additional dependencies which are necessary for > > the builds of Digikam and SubtitleComposer to commence on the CI > > system for Windows. These failures by PIM have therefore indirectly > > harmed other KDE projects. > > > > As this is not the first time that PIM has caused issues in this > > manner, I would therefore like to proceed with blacklisting PIM from > > the CI system. This would include prohibiting other projects from > > depending on any part of PIM, so those projects that have a required > > dependency on PIM would also have their builds removed by this. > > Of course stuff tends to break more often when you look at 50+ actively > developed repos compared to a single barely alive one. And yes, I do very well > understand that this can be frustrating. > > > Whilst I would prefer another solution to this, given that it is a > > recurring issue that makes maintenance of the CI system substantially > > harder, I see the removal of PIM from the equation as the only > > reasonable path forward. > > > > Should the PIM developers wish to avoid this consequence for their > > actions, they will need to provide an action plan as to how this will > > be avoided in the future. > > I cannot realistically promise more than what I am already doing on this (as > outlined above), the combination of me not able to pay enough attention to the > CI state and things blowing up in multiple repos on the same day is very rare > though. If that's not enough, others need to help here as well. > > > Fixing the current set of failures will not prevent this blacklisting > > action from being carried out - as a recurring issue it needs a > > permanent solution. > > What do you envision this permanent solution to look like? It's hard to
Re: I need help fixing krita's flatpak build on binary factory
On Wed, Dec 4, 2019 at 3:12 AM Boudewijn Rempt wrote: > > Krita's flatpak nightly and nightly stable builds are broken, and I don't > know why. It's almost as if flatpak-builder tries to parse the yaml file as > json: > > [2019-12-02T22:37:51.554Z] + flatpak-builder --force-clean > --delete-build-dirs --arch=x86_64 --ccache --sandbox --user > --install-deps-from=flathub --repo=/home/packaging/staging-repo/ > --subject=Built on Mon Dec 2 23:37:51 CET 2019 app > /home/packaging/jenkins/workspace/Krita_nightly_flatpak/packaging/linux/flatpak/org.kde.krita-nightly.yaml > [2019-12-02T22:37:51.554Z] Can't parse > '/home/packaging/jenkins/workspace/Krita_nightly_flatpak/packaging/linux/flatpak/org.kde.krita-nightly.yaml': > :1:6: Parse error: unexpected identifier `app-id', expected value > > Was there a recent change that disabled support for yaml files and should > those be rewritten as json files? November 16 would roughly line up with when we changed the way the builders work, so it's possible that the version of flatpak-builder in use was changed at that time. Looking in that folder, I do see a .json format version which is 3 months out of date. Maybe removing it will help? > -- > Boudewijn Rempt | https://www.valdyas.org | https://www.krita.org Cheers, Ben
Re: I need help fixing krita's flatpak build on binary factory
On Thu, Dec 5, 2019 at 1:25 PM Aleix Pol wrote: > > On Wed, Dec 4, 2019 at 9:00 AM Ben Cooksley wrote: > > > > On Wed, Dec 4, 2019 at 3:12 AM Boudewijn Rempt wrote: > > > > > > Krita's flatpak nightly and nightly stable builds are broken, and I don't > > > know why. It's almost as if flatpak-builder tries to parse the yaml file > > > as json: > > > > > > [2019-12-02T22:37:51.554Z] + flatpak-builder --force-clean > > > --delete-build-dirs --arch=x86_64 --ccache --sandbox --user > > > --install-deps-from=flathub --repo=/home/packaging/staging-repo/ > > > --subject=Built on Mon Dec 2 23:37:51 CET 2019 app > > > /home/packaging/jenkins/workspace/Krita_nightly_flatpak/packaging/linux/flatpak/org.kde.krita-nightly.yaml > > > [2019-12-02T22:37:51.554Z] Can't parse > > > '/home/packaging/jenkins/workspace/Krita_nightly_flatpak/packaging/linux/flatpak/org.kde.krita-nightly.yaml': > > > :1:6: Parse error: unexpected identifier `app-id', expected value > > > > > > Was there a recent change that disabled support for yaml files and should > > > those be rewritten as json files? > > > > November 16 would roughly line up with when we changed the way the > > builders work, so it's possible that the version of flatpak-builder in > > use was changed at that time. > > > > Looking in that folder, I do see a .json format version which is 3 > > months out of date. > > Maybe removing it will help? > > Hi Ben, > It's quite weird, I looked into it few days ago and it built properly > on my system. It feels to me like flatpak-builder would be using the > json parser to do yaml? > > Anyhow, maybe it could make sense updating the flatpak-builder version > on these systems? It's even possible to run a flatpak version of > flatpak-builder. I can help with that I just am not sure where to look > at. root@ange ~ # flatpak-builder --version flatpak-builder 0.10.9 At the moment there is only one machine, but I do intend to make it three in the end. (Once all the dust is settled for Windows/FreeBSD, and we have everything in a stable enough condition that I can actually do it of course) > > Aleix Cheers, Ben
Re: Latte Dock on FreeBSD
On Thu, Jan 2, 2020 at 9:39 AM Michail Vourlakos wrote: > > https://phabricator.kde.org/source/latte-dock/browse/master/app/FakeTarget.cmake > > Can you please make a PR because I am not that sure I understand what I need > to change in it Sorry, CMake is a bit beyond me in this case. We would be wanting to change line 16 in this case, to see if ECM_ENABLE_SANITIZERS is empty (because if it isn't, we shouldn't be running qmllint). Does anyone on the development list know of the best way to tackle this? (Background to this: qmllint is a application shipped with Qt, and thus not built with ASAN, but our libraries are built with ASAN so trying to run qmllint on the CI system will always fail. You can workaround this on Linux by forcibly injecting ASAN into the process using LD_PRELOAD but ASAN is statically linked on FreeBSD, so it is impossible to workaround there) Cheers, Ben > > Στις Τετ, 1 Ιαν 2020, 10:30 μ.μ. ο χρήστης Ben Cooksley > έγραψε: >> >> On Thu, Jan 2, 2020 at 7:25 AM Michail Vourlakos >> wrote: >> > >> > found in the Internet that the CMake command should contain: >> > >> > >> > cmake ... -DCMAKE_BUILD_TYPE=Release >> > >> >> Because we are running a CI system, and therefore want tests with full >> asserts enabled, we use a build type of Debug. >> I haven't seen the qmllint failures elsewhere, mind linking me to >> where to find app/FakeTarget.cmake? >> >> If it is part of Latte Dock, you'll need to change it to skip qmllint >> when ASAN support is enabled (which is controlled by >> -DECM_ENABLE_SANITIZERS being passed to CMake) >> >> Cheers, >> Ben >> >> > >> > Στις Τετ, 1 Ιαν 2020 στις 8:16 μ.μ., ο/η Ben Cooksley >> > έγραψε: >> >> >> >> Hi Michail, >> >> >> >> A while back when we upgraded things on FreeBSD, one of the >> >> unfortunate casualties of this process was the Latte Dock builds on >> >> that platform, which now fail with a ASAN related error. >> >> >> >> (See >> >> https://build.kde.org/job/Extragear/job/latte-dock/job/stable-kf5-qt5%20FreeBSDQt5.13/99/console) >> >> >> >> Examining the build log, I note that it looks like you are running >> >> `qmllint` as part of the final steps before linking the executable. >> >> >> >> Could you confirm whether this is the case? >> >> >> >> Cheers, >> >> Ben
Re: Latte Dock on FreeBSD
On Sat, Jan 4, 2020 at 10:54 AM Albert Astals Cid wrote: > > El divendres, 3 de gener de 2020, a les 10:01:50 CET, Ben Cooksley va > escriure: > > On Thu, Jan 2, 2020 at 9:39 AM Michail Vourlakos > > wrote: > > > > > > https://phabricator.kde.org/source/latte-dock/browse/master/app/FakeTarget.cmake > > > > > > Can you please make a PR because I am not that sure I understand what I > > > need to change in it > > > > Sorry, CMake is a bit beyond me in this case. > > We would be wanting to change line 16 in this case, to see if > > ECM_ENABLE_SANITIZERS is empty (because if it isn't, we shouldn't be > > running qmllint). > > > > Does anyone on the development list know of the best way to tackle this? > > > > (Background to this: qmllint is a application shipped with Qt, and > > thus not built with ASAN, but our libraries are built with ASAN so > > trying to run qmllint on the CI system will always fail. You can > > workaround this on Linux by forcibly injecting ASAN into the process > > using LD_PRELOAD but ASAN is statically linked on FreeBSD, so it is > > impossible to workaround there) > > You mean there's no dynamic library ASAN at all in FreeBSD? That is correct, ASAN is statically linked on FreeBSD. I'm not sure if that is because of the way they've decided to package ASAN, or if it because that is the way that Clang does it. I'm inclined to think it is Clang, because even on Linux/GCC if you try to run an executable that isn't linked to ASAN, but which does use a library that is linked to ASAN, then it will abort on you with a message about ASAN needing to be loaded first (before anything else is loaded) That is why Kirigami and Marble have the "force-inject-asan" flag set in their Build Specifications on the CI system to make their tests pass - because Kirigami reiies on tools shipped with Qt itself (and we use distribution Qt), and Marble doesn't use ECM (so the necessary flags never get passed to the compiler - Marble also doesn't build on FreeBSD due to this). We don't set that flag globally because Skrooge relies on Java for some of it's unit tests, and Java really does not like ASAN being injected into it (it crashes). > > Cheers, > Albert Cheers, Ben > > > > > Cheers, > > Ben > > > > > > > > Στις Τετ, 1 Ιαν 2020, 10:30 μ.μ. ο χρήστης Ben Cooksley > > > έγραψε: > > >> > > >> On Thu, Jan 2, 2020 at 7:25 AM Michail Vourlakos > > >> wrote: > > >> > > > >> > found in the Internet that the CMake command should contain: > > >> > > > >> > > > >> > cmake ... -DCMAKE_BUILD_TYPE=Release > > >> > > > >> > > >> Because we are running a CI system, and therefore want tests with full > > >> asserts enabled, we use a build type of Debug. > > >> I haven't seen the qmllint failures elsewhere, mind linking me to > > >> where to find app/FakeTarget.cmake? > > >> > > >> If it is part of Latte Dock, you'll need to change it to skip qmllint > > >> when ASAN support is enabled (which is controlled by > > >> -DECM_ENABLE_SANITIZERS being passed to CMake) > > >> > > >> Cheers, > > >> Ben > > >> > > >> > > > >> > Στις Τετ, 1 Ιαν 2020 στις 8:16 μ.μ., ο/η Ben Cooksley > > >> > έγραψε: > > >> >> > > >> >> Hi Michail, > > >> >> > > >> >> A while back when we upgraded things on FreeBSD, one of the > > >> >> unfortunate casualties of this process was the Latte Dock builds on > > >> >> that platform, which now fail with a ASAN related error. > > >> >> > > >> >> (See > > >> >> https://build.kde.org/job/Extragear/job/latte-dock/job/stable-kf5-qt5%20FreeBSDQt5.13/99/console) > > >> >> > > >> >> Examining the build log, I note that it looks like you are running > > >> >> `qmllint` as part of the final steps before linking the executable. > > >> >> > > >> >> Could you confirm whether this is the case? > > >> >> > > >> >> Cheers, > > >> >> Ben > > > > > >
Re: Latte Dock on FreeBSD
On Sat, Jan 4, 2020 at 11:40 PM Albert Astals Cid wrote: > > El dissabte, 4 de gener de 2020, a les 8:02:54 CET, Ben Cooksley va escriure: > > On Sat, Jan 4, 2020 at 10:54 AM Albert Astals Cid wrote: > > > > > > El divendres, 3 de gener de 2020, a les 10:01:50 CET, Ben Cooksley va > > > escriure: > > > > On Thu, Jan 2, 2020 at 9:39 AM Michail Vourlakos > > > > wrote: > > > > > > > > > > https://phabricator.kde.org/source/latte-dock/browse/master/app/FakeTarget.cmake > > > > > > > > > > Can you please make a PR because I am not that sure I understand what > > > > > I need to change in it > > > > > > > > Sorry, CMake is a bit beyond me in this case. > > > > We would be wanting to change line 16 in this case, to see if > > > > ECM_ENABLE_SANITIZERS is empty (because if it isn't, we shouldn't be > > > > running qmllint). > > > > > > > > Does anyone on the development list know of the best way to tackle this? > > > > > > > > (Background to this: qmllint is a application shipped with Qt, and > > > > thus not built with ASAN, but our libraries are built with ASAN so > > > > trying to run qmllint on the CI system will always fail. You can > > > > workaround this on Linux by forcibly injecting ASAN into the process > > > > using LD_PRELOAD but ASAN is statically linked on FreeBSD, so it is > > > > impossible to workaround there) > > > > > > You mean there's no dynamic library ASAN at all in FreeBSD? > > > > That is correct, ASAN is statically linked on FreeBSD. > > Just being very verborse here, the fact that the library is statically linked > when compiling doesn't mean that a dynamic version can not also exist, i.e. > on my system i have both libz.a and libz.so Yes. The dynamic version does not exist at all to my knowledge on FreeBSD. > > > I'm not sure if that is because of the way they've decided to package > > ASAN, or if it because that is the way that Clang does it. > > > > I'm inclined to think it is Clang, because even on Linux/GCC if you > > try to run an executable that isn't linked to ASAN, but which does use > > a library that is linked to ASAN, then it will abort on you with a > > message about ASAN needing to be loaded first (before anything else is > > loaded) > > > > That is why Kirigami and Marble have the "force-inject-asan" flag set > > in their Build Specifications on the CI system to make their tests > > pass - because Kirigami reiies on tools shipped with Qt itself (and we > > use distribution Qt), and Marble doesn't use ECM (so the necessary > > flags never get passed to the compiler - Marble also doesn't build on > > FreeBSD due to this). > > If we have confirmed with the FreeBSD people that there's no way to get a > libasan.so we can inject with LD_PRELOAD I only see two options: > * Build Qt with ASAN > * Don't build anything with ASAN in FreeBSD Not sure what the line of thinking is here? Apart from those two exceptions, KDE projects aren't affected by this as they don't rely on Qt executables that load KDE libraries - the only time this really happens is with some QML stuff (which is why Kirigami is affected). Marble is simply an abberation as it does not use ECM yet does use KDE libraries. The rest all use ECM, and are thus built with ASAN enabled, which ensures that they not only build successfully, but are also able to be executed successfully at runtime (a situation we cannot guarantee on Linux - hence why Kirigami/Marble need the forced injection) The only thing having a libasan.so would give us is passing tests for Kirigami. > > The first may be a huge can of worms though since maybe we need to build > other stuff that Qt depends on with ASAN, so i guess option two is the > solution? > > It wouldn't be very terrible since AFAIK we don't have many "FreeBSD only" > code, or do we? > > Cheers, > Albert > Cheers, Ben > > > > We don't set that flag globally because Skrooge relies on Java for > > some of it's unit tests, and Java really does not like ASAN being > > injected into it (it crashes). > > > > > > > > Cheers, > > > Albert > > > > Cheers, > > Ben > > > > > > > > > > > > > Cheers, > > > > Ben > > > > > > > > > > > > > > Στις Τετ, 1 Ιαν 2020, 10:30 μ.μ. ο χρήστης Ben Cooksley > > > >
Re: Latte Dock on FreeBSD
On Mon, Jan 6, 2020 at 10:31 AM Albert Astals Cid wrote: > > El dissabte, 4 de gener de 2020, a les 19:26:02 CET, Ben Cooksley va escriure: > > On Sat, Jan 4, 2020 at 11:40 PM Albert Astals Cid wrote: > > > > > > El dissabte, 4 de gener de 2020, a les 8:02:54 CET, Ben Cooksley va > > > escriure: > > > > On Sat, Jan 4, 2020 at 10:54 AM Albert Astals Cid wrote: > > > > > > > > > > El divendres, 3 de gener de 2020, a les 10:01:50 CET, Ben Cooksley va > > > > > escriure: > > > > > > On Thu, Jan 2, 2020 at 9:39 AM Michail Vourlakos > > > > > > wrote: > > > > > > > > > > > > > > https://phabricator.kde.org/source/latte-dock/browse/master/app/FakeTarget.cmake > > > > > > > > > > > > > > Can you please make a PR because I am not that sure I understand > > > > > > > what I need to change in it > > > > > > > > > > > > Sorry, CMake is a bit beyond me in this case. > > > > > > We would be wanting to change line 16 in this case, to see if > > > > > > ECM_ENABLE_SANITIZERS is empty (because if it isn't, we shouldn't be > > > > > > running qmllint). > > > > > > > > > > > > Does anyone on the development list know of the best way to tackle > > > > > > this? > > > > > > > > > > > > (Background to this: qmllint is a application shipped with Qt, and > > > > > > thus not built with ASAN, but our libraries are built with ASAN so > > > > > > trying to run qmllint on the CI system will always fail. You can > > > > > > workaround this on Linux by forcibly injecting ASAN into the process > > > > > > using LD_PRELOAD but ASAN is statically linked on FreeBSD, so it is > > > > > > impossible to workaround there) > > > > > > > > > > You mean there's no dynamic library ASAN at all in FreeBSD? > > > > > > > > That is correct, ASAN is statically linked on FreeBSD. > > > > > > Just being very verborse here, the fact that the library is statically > > > linked when compiling doesn't mean that a dynamic version can not also > > > exist, i.e. on my system i have both libz.a and libz.so > > > > Yes. The dynamic version does not exist at all to my knowledge on FreeBSD. > > > > > > > > > I'm not sure if that is because of the way they've decided to package > > > > ASAN, or if it because that is the way that Clang does it. > > > > > > > > I'm inclined to think it is Clang, because even on Linux/GCC if you > > > > try to run an executable that isn't linked to ASAN, but which does use > > > > a library that is linked to ASAN, then it will abort on you with a > > > > message about ASAN needing to be loaded first (before anything else is > > > > loaded) > > > > > > > > That is why Kirigami and Marble have the "force-inject-asan" flag set > > > > in their Build Specifications on the CI system to make their tests > > > > pass - because Kirigami reiies on tools shipped with Qt itself (and we > > > > use distribution Qt), and Marble doesn't use ECM (so the necessary > > > > flags never get passed to the compiler - Marble also doesn't build on > > > > FreeBSD due to this). > > > > > > If we have confirmed with the FreeBSD people that there's no way to get a > > > libasan.so we can inject with LD_PRELOAD I only see two options: > > > * Build Qt with ASAN > > > * Don't build anything with ASAN in FreeBSD > > > > Not sure what the line of thinking is here? > > The line of thinking was "we can't [always] make it work, so don't do it". > > I know it works in 99.99% of the cases but if it doesn't work in 0.01% it > means it doesn't work at all, no? I can see where you are coming from here yes. The outcome of that would be that we disable ASAN on FreeBSD. > > Cheers, > Albert Cheers, Ben > > > > > Apart from those two exceptions, KDE projects aren't affected by this > > as they don't rely on Qt executables that load KDE libraries - the > > only time this really happens is with some QML stuff (which is why > > Kirigami is affected). Marble is simply an abberation as it does not > > use ECM yet does use KDE librar
Get Hot New Stuff - Legacy Endpoints and Multiple Requests (Discover?)
Hi all, While diagnosing an issue this evening with cdn.kde.org, I noticed that we are still getting an extremely large number of requests for the legacy OCS/GHNS providers.xml endpoint, which is supposed to only exist for compatibility with older applications. Looking on LXR i've found that a substantial number of our applications are still using the old url in their most recent releases (see https://lxr.kde.org/search?_filestring=&_string=%2Focs%2Fproviders.xml) All new code should be using autoconfig.kde.org now, and all old code should be transitioning to using the new url. The legacy url is significantly more resource intensive for our servers to support and our ability to scale this endpoint is substantially more limited when compared with the new endpoint so it is important that this is moved over (to the extent Enterprise/Long term LTS distributions should be encouraged to patch this) Additionally, I believe there to be a multiple request bug somewhere as this endpoint is being hit approximately 3-4 times in the same second for each client we see. Based on the corresponding hit to autoconfig.kde.org/discover/featured-5.9.json which happens at the same time it is likely the application showing this fault the most is Discover. If we could please get the above all fixed up that would be appreciated. Thanks, Ben
Re: Get Hot New Stuff - Legacy Endpoints and Multiple Requests (Discover?)
On Fri, Jan 31, 2020 at 1:05 AM Michael Reeves wrote: > > There seems to be some sort of ssl certificate issue with autoconfig.kde.org. > On kubuntu this triggers a warning as soon as discover starts that may be a > contributing factor. I've checked and can find no fault with autoconfig.kde.org. Both wget and kioclient5 raise no objections on my system, and both Chrome and Firefox are perfectly happy with it as well. The Qualys SSL Server Test also confirms that we have no setup issues (aside from including the root certificate in the chain we send, which is harmless). The results of that test can be found at https://www.ssllabs.com/ssltest/analyze.html?d=autoconfig.kde.org I'm afraid the fault is either with Kubuntu (which I highly doubt) or your local network (much more probable) in this instance. Should you believe the fault is on our side I will need information from the "Details" pane as to why KIO refuses to trust connections to autoconfig.kde.org on your system. Cheers, Ben > > On Thu, Jan 30, 2020, 4:20 AM Ben Cooksley wrote: >> >> Hi all, >> >> While diagnosing an issue this evening with cdn.kde.org, I noticed >> that we are still getting an extremely large number of requests for >> the legacy OCS/GHNS providers.xml endpoint, which is supposed to only >> exist for compatibility with older applications. >> >> Looking on LXR i've found that a substantial number of our >> applications are still using the old url in their most recent releases >> (see https://lxr.kde.org/search?_filestring=&_string=%2Focs%2Fproviders.xml) >> >> All new code should be using autoconfig.kde.org now, and all old code >> should be transitioning to using the new url. >> >> The legacy url is significantly more resource intensive for our >> servers to support and our ability to scale this endpoint is >> substantially more limited when compared with the new endpoint so it >> is important that this is moved over (to the extent Enterprise/Long >> term LTS distributions should be encouraged to patch this) >> >> Additionally, I believe there to be a multiple request bug somewhere >> as this endpoint is being hit approximately 3-4 times in the same >> second for each client we see. Based on the corresponding hit to >> autoconfig.kde.org/discover/featured-5.9.json which happens at the >> same time it is likely the application showing this fault the most is >> Discover. >> >> If we could please get the above all fixed up that would be appreciated. >> >> Thanks, >> Ben
Banning QNetworkAccessManager
Hi all, For an extremely long time now, it has been a known issue with the QNetworkAccessManager that by default it does not follow redirects as issued by the server it is accessing. This is something the Qt Project has refused to address using the justification of 'behavioural compatibility' This behaviour is contrary to that of just about every other HTTP stack (with the exception of libcurl from my understanding) and is behaviour that nobody expects. As a consequence of this (broken) behaviour, Sysadmin has been effectively forced to implement workarounds and other compatibility measures in place to keep applications functional. These measures harm the long term maintainability of our systems, and sometimes limit our ability to make services more scalable. In some instances, the cost of compatibility measures would be too high, resulting in functionality of applications being broken. I am aware of at least one instance where if a compatibility measure we maintain is removed, the application will crash (segfault) during it's operation (a failure that makes the application unusable) Prior to now, i've taken the approach of advertising that QNetworkAccessManager is broken and needs a flag set within applications when used, however unfortunately this seems to have been an ineffective approach and cases of broken behaviour continue to appear (despite this brokeness being known about and advertised since back in the Qt 4.x series when this class was introduced) Therefore, given the Qt Project is unlikely to change their position on this, I would like to propose the following: 1) That effective immediately, QNetworkAccessManager and it's children classes be banned and prohibited within KDE software, to be enforced by server side hooks. 2) That we fork QNetworkAccessManager and the associated classes within the appropriate Framework (to be determined), where the defective behaviour can then be corrected. Comments? Regards, Ben Cooksley KDE Sysadmin
Re: Banning QNetworkAccessManager
On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > I agree on the problem of QNAM's default, see also https://conf.kde.org/en/ > akademy2019/public/events/135 on that subject. > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > [...] > > Prior to now, i've taken the approach of advertising that > > QNetworkAccessManager is broken and needs a flag set within > > applications when used, however unfortunately this seems to have been > > an ineffective approach and cases of broken behaviour continue to > > appear (despite this brokeness being known about and advertised since > > back in the Qt 4.x series when this class was introduced) > > > > Therefore, given the Qt Project is unlikely to change their position > > on this, I would like to propose the following: > > The Qt Project is actually in favor of changing at least the redirection > default to exactly what we want it to be. > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, and got > approval both by Lars and one of the maintainers. It is merely stuck in > dealing with the unit test fallout in Qt's CI that somebody has to take care > of. Doesn't immediately help us of course, but claiming Qt is unwilling to > change this is simply wrong. Last I heard they were unwilling to change the default, so it's nice to know they're looking at revisiting their error. Back when this issue originally developed back in the Qt 4.x era (just a couple of releases after QNAM appeared) the Qt Developers at the time not only refused to change the default, but also refused to consider adding any kind of convenience function. They considered that their position was the most correct one and that following redirects simply wasn't required (as it isn't a MUST in the RFCs). It was later on (around Qt 5.7 I think) that a set of functions and enums were added to Qt to make it convenient to change the behaviour. When these were initially introduced the behaviour was changed to follow redirects, however that was then overturned before it was released (on the basis of an absolute need for "behavioural compatibility", something that wasn't an issue when the change went through initially, so my suspicion is a customer of Digia at the time threw their toys) I wasn't involved directly in the conversations in the Qt 5.7 timeframe, so don't have a link to that i'm afraid. The Qt 4.x one was a conversation on IRC. While that merge request is promising, it unfortunately is targeted at the 'dev' branch, which if i'm not wrong is Qt 6.x, so it will be some time before we get to see the benefit of that (more on that below). > > > 1) That effective immediately, QNetworkAccessManager and it's children > > classes be banned and prohibited within KDE software, to be enforced > > by server side hooks. > > 2) That we fork QNetworkAccessManager and the associated classes > > within the appropriate Framework (to be determined), where the > > defective behaviour can then be corrected. > > While this might solve the redirection problem, it brings us yet another then > unmaintained HTTP implementation next to the one in KIO, which is a far bigger > problem long term. We need less of those, not more. > It is not ideal I agree. What I would like to ensure however is that we are not impacted in the future by any policy decisions made by the Qt Developers in respect of this set of classes. > To address the problem of people not using the correct defaults, I did propose > a much less extreme approach in the above mentioned talk at Akademy, namely > having a KNetworkAccessManager as a sub-class of QNAM in a low-tier frameworks > that essentially just enables redirects and HSTS. QNAM's implementation isn't > the problem after all, the defaults are. > This may work, but we would need to force people to move to using the wrapper, and it still leaves us exposed to anywhere that retrieves a QNAM constructed by code located elsewhere. > Commit hooks warning about QNAM usage might still be a good idea, but as a > warning only. Hard blocking QNAM-using code would block at least three repos I > spend a lot of time on currently, none of which even talks to KDE servers. > > If you need to take more drastic measures against code not doing this > correctly regardless, you can do that my dropping the server-side workarounds. > Yes, that would break the affected application possibly, but at least it would > not cause massive collateral damage for the people using this correctly. > Nicolas did a spot check of some of the places where QNAM is used in KDE, and at first glance it appears that not a single one enabled the following of redirects :( It is likely that very few places
Re: Banning QNetworkAccessManager
On Mon, Feb 3, 2020 at 10:49 PM David Edmundson wrote: > > I updated: > > https://community.kde.org/Policies/API_to_Avoid > > Which had no mention of this. Many thanks for updating the wiki David. > > David Cheers, Ben
Re: Banning QNetworkAccessManager
On Mon, Feb 3, 2020 at 11:51 PM Johan Ouwerkerk wrote: > > On Mon, Feb 3, 2020 at 11:27 AM laurent Montel wrote: > > > > Le lundi 3 février 2020, 10:49:10 CET David Edmundson a écrit : > > > I updated: > > > > > > https://community.kde.org/Policies/API_to_Avoid > > > > > > Which had no mention of this. > > > > > > David > > > > I think that you made an error > > > > "networkAccessManger->setAttribute(QNetworkRequest::FollowRedirectsAttribute, > > true); " > > Doesn't exist it's a enum from QnetworkRequest::RedirectPolicy > > And FollowRedirectsAttribute is old value > > It seems that we need to use QnetworkRequest::NoLessSafeRedirectPolicy > > directly no ? > > > > Yes, the example code is definitely wrong: in the real world redirects > are an attack vector. A few cases to consider: > > * Loops of redirects (could happen if the site is broken) > * Leaking sensitive information via e.g. the Referrer header I would recommend follwoing the various browsers (Firefox & Chromium in particular) behaviour here. Not only are they what we generally test redirects with when doing server maintenance, but they also have put quite a bit of work into being secure. With regards to loops, stopping after 10 or so redirects should be more than sufficient as a safe guard here. 5 if you'd like to be a bit more restrictive. I can't imagine a need with our systems to need more than 3 (first to https, second from somewhere to the mirror network redirector, third to a mirror) Because mirrors often don't support https (and Mirrorbrain doesn't support handling https mirrors) clients that interact with download.kde.org or files.kde.org *must* support transitioning from https:// to http:// (under a different domain - so https://download.kde.org to http://ftp.gwdg.de for instance). This would primarily affected applications that need to fetch large binary assets. > > Regards, > > - Johan Cheers, Ben
Re: Review Kid3
On Thu, Feb 13, 2020 at 9:12 PM Christoph Feck wrote: > > On 02/13/20 07:11, Urs Fleisch wrote: > > As you may know, Kid3, an audio tagger, has moved from SourceForge.net to > > kde.org. Jonathan Riddell is attending the incubation process as a sponsor. > > To complete the incubation, I would kindly ask you to execute a KDE review > > on the project. > > > > Website: https://kid3.kde.org/ > > Git, Issues: https://invent.kde.org/kde/kid3/ > > Is there a way to subscribe to bugs reported there? By starring the project. Once that is done, you can customise the address the notifications are delivered to, along with how the notifications are delivered to you, at https://invent.kde.org/profile/notifications Please note that in general, Gitlab is supposed to be used to handle Developer tasks/issues, rather than user bugs. > > > Incubator: https://community.kde.org/Incubator/Projects/kid3 > > Translation state: https://l10n.kde.org/stats/gui/trunk-kf5/po/kid3_qt.po/ > > Documentation state: https://l10n.kde.org/stats/doc/trunk-kf5/po/kid3.po/ > > > > Kid3 exists since 2003 and I would say that it is quite mature. As it is > > also available as a Qt-only version (without KDE dependencies) and for > > macOS, Windows and Android, the build system differs a bit from the typical > > KDE application. > > -- > Christoph Feck > KDE Bug Triaging Team > Cheers, Ben
Re: Banning QNetworkAccessManager
On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > I agree on the problem of QNAM's default, see also https://conf.kde.org/en/ > akademy2019/public/events/135 on that subject. > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > [...] > > Prior to now, i've taken the approach of advertising that > > QNetworkAccessManager is broken and needs a flag set within > > applications when used, however unfortunately this seems to have been > > an ineffective approach and cases of broken behaviour continue to > > appear (despite this brokeness being known about and advertised since > > back in the Qt 4.x series when this class was introduced) > > > > Therefore, given the Qt Project is unlikely to change their position > > on this, I would like to propose the following: > > The Qt Project is actually in favor of changing at least the redirection > default to exactly what we want it to be. > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, and got > approval both by Lars and one of the maintainers. It is merely stuck in > dealing with the unit test fallout in Qt's CI that somebody has to take care > of. Doesn't immediately help us of course, but claiming Qt is unwilling to > change this is simply wrong. > > > 1) That effective immediately, QNetworkAccessManager and it's children > > classes be banned and prohibited within KDE software, to be enforced > > by server side hooks. > > 2) That we fork QNetworkAccessManager and the associated classes > > within the appropriate Framework (to be determined), where the > > defective behaviour can then be corrected. > > While this might solve the redirection problem, it brings us yet another then > unmaintained HTTP implementation next to the one in KIO, which is a far bigger > problem long term. We need less of those, not more. > > To address the problem of people not using the correct defaults, I did propose > a much less extreme approach in the above mentioned talk at Akademy, namely > having a KNetworkAccessManager as a sub-class of QNAM in a low-tier frameworks > that essentially just enables redirects and HSTS. QNAM's implementation isn't > the problem after all, the defaults are. > > Commit hooks warning about QNAM usage might still be a good idea, but as a > warning only. Hard blocking QNAM-using code would block at least three repos I > spend a lot of time on currently, none of which even talks to KDE servers. > > If you need to take more drastic measures against code not doing this > correctly regardless, you can do that my dropping the server-side workarounds. > Yes, that would break the affected application possibly, but at least it would > not cause massive collateral damage for the people using this correctly. > > It would also help to know where specifically we have that problem, so we can > actually solve it, and so we can figure out why we failed to fix this there > earlier. Just bringing this up again - it seems we've not had much movement on this aside from the Wiki page. Examining that Qt merge request, it not only is targeted at just Qt 6.x, it also hasn't had any movement since the start of October 2019 - 4 months ago. Given that we are going to be on Qt 5.x for some time, that merge request can't be considered to be the 'fix' for this issue. We need a solution that will ensure software released today doesn't cause us pain in several years time, because as I illustrated in an earlier email to this thread, software has a habit of remaining in use for an extremely long amount of time (August 2014 being the release date of the Marble versions in question hitting that old URL). The easiest path forward is to simply ban QNetworkAccessManager. > > Regards, > Volker Regards, Ben
Re: Banning QNetworkAccessManager
On Wed, Feb 19, 2020 at 9:30 PM Volker Krause wrote: > > On Wednesday, 19 February 2020 08:05:01 CET Ben Cooksley wrote: > > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > > I agree on the problem of QNAM's default, see also > > > https://conf.kde.org/en/ > > > akademy2019/public/events/135 on that subject. > > > > > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > > > [...] > > > > > > > Prior to now, i've taken the approach of advertising that > > > > QNetworkAccessManager is broken and needs a flag set within > > > > applications when used, however unfortunately this seems to have been > > > > an ineffective approach and cases of broken behaviour continue to > > > > appear (despite this brokeness being known about and advertised since > > > > back in the Qt 4.x series when this class was introduced) > > > > > > > > Therefore, given the Qt Project is unlikely to change their position > > > > > > > on this, I would like to propose the following: > > > The Qt Project is actually in favor of changing at least the redirection > > > default to exactly what we want it to be. > > > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, and > > > got approval both by Lars and one of the maintainers. It is merely stuck > > > in dealing with the unit test fallout in Qt's CI that somebody has to > > > take care of. Doesn't immediately help us of course, but claiming Qt is > > > unwilling to change this is simply wrong. > > > > > > > 1) That effective immediately, QNetworkAccessManager and it's children > > > > classes be banned and prohibited within KDE software, to be enforced > > > > by server side hooks. > > > > 2) That we fork QNetworkAccessManager and the associated classes > > > > within the appropriate Framework (to be determined), where the > > > > defective behaviour can then be corrected. > > > > > > While this might solve the redirection problem, it brings us yet another > > > then unmaintained HTTP implementation next to the one in KIO, which is a > > > far bigger problem long term. We need less of those, not more. > > > > > > To address the problem of people not using the correct defaults, I did > > > propose a much less extreme approach in the above mentioned talk at > > > Akademy, namely having a KNetworkAccessManager as a sub-class of QNAM in > > > a low-tier frameworks that essentially just enables redirects and HSTS. > > > QNAM's implementation isn't the problem after all, the defaults are. > > > > > > Commit hooks warning about QNAM usage might still be a good idea, but as a > > > warning only. Hard blocking QNAM-using code would block at least three > > > repos I spend a lot of time on currently, none of which even talks to KDE > > > servers. > > > > > > If you need to take more drastic measures against code not doing this > > > correctly regardless, you can do that my dropping the server-side > > > workarounds. Yes, that would break the affected application possibly, but > > > at least it would not cause massive collateral damage for the people > > > using this correctly. > > > > > > It would also help to know where specifically we have that problem, so we > > > can actually solve it, and so we can figure out why we failed to fix this > > > there earlier. > > > > Just bringing this up again - it seems we've not had much movement on > > this aside from the Wiki page. > > > > Examining that Qt merge request, it not only is targeted at just Qt > > 6.x, it also hasn't had any movement since the start of October 2019 - > > 4 months ago. > > Given that we are going to be on Qt 5.x for some time, that merge > > request can't be considered to be the 'fix' for this issue. > > The targeting of Qt6 is due to this aiming at dev when that was still Qt 5.x, > but you are right of course, somebody needs to do the work there to drive this > to completion, no matter in which version it will actually land. Indeed. > > > We need a solution that will ensure software released today doesn't > > cause us pain in several years time, because as I illustrated in an > > earlier email to this thread, software has a habit of remaining in use > > for an extremely long amount of time (August 2014 being the release > > date of the Marble versions in question hittin
Re: Fixing QNetworkAccessManager use
On Thu, Feb 20, 2020 at 9:58 AM Friedrich W. H. Kossebau wrote: > > Am Mittwoch, 19. Februar 2020, 21:01:20 CET schrieb Johan Ouwerkerk: > > On Wed, Feb 19, 2020 at 2:09 PM Friedrich W. H. Kossebau > > > > wrote: > > > Personally I am still unsure what the actual issue is. Why are redirects > > > needed at all. Why all the address changes all the time? > > > > It is part of the HTTP spec for servers to be able to inform clients > > that resource /foo/bar has moved to /bar/baz, either temporarily or > > permanently. > > :) Thanks for that explanation, but that was not my question here (that part I > am well aware of, done my share of web stuff). > > It was rather: why are subdomain names and/or access paths not once properly > designed, but instead changed so often that redirection seems so important to > be a default feature? Just because one can? Things don't change extremely often. Sometimes however requirements or other factors change, which necessitates changing where a resource is hosted. When this happens, it is extremely useful to have the ability to relocate it elsewhere. To use an example, when we first setup files.kde.org it was used by a couple of things, including Necessitas for the Qt binaries that get downloaded on to end user (Android) devices. When this was first established, traffic was well within the reasonable bounds we had expected when setting this up, and everything was served directly by our (single) server. This went quite well for a while. Sometime a bit later, an application was released on Google Play that used Necessitas which was *extremely* popular, to the extent it caused around a terabyte of data to be used within 48 hours or so. Hetzner bandwidth was at this time not only limited to 100mbps, but also capped - with the limit being 5 TB per month and overage after that resulting in a charge per terabyte. We therefore made the decision to convert files.kde.org to a mirror network (like was already in place for download.kde.org), with redirection taking place using Mirrorbrain. We were able to complete this transition quickly thanks to the generous support of some of our mirrors who established mirrors of files.kde.org. Fortunately Necessitas had full support for handling redirects, so this is something we were able to accomplish without any issues. Had redirect support not been available, we would have been left with no way out at that time. I also have other examples involving Marble (including where we got bitten by QNetworkAccessManager for the very first time - back in January 2012) and numerous other KDE Edu applications (all of which fortunately avoided QNAM). > When we write code, we try to keep API stable as much as possible, and only > change API when really useful, and that means for the consumer. When doing > references in text we try to have eternally stable pointers (thanks ISBN & > Co.), > > But this request for stable URLs on the internet might be an idealistic fight > against windmills of a web 1.0 person... > > Cheers > Friedrich > > Cheers, Ben
Re: Fixing QNetworkAccessManager use
On Thu, Feb 20, 2020 at 2:09 AM Friedrich W. H. Kossebau wrote: > > Am Mittwoch, 19. Februar 2020, 08:05:01 CET schrieb Ben Cooksley: > > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > > It would also help to know where specifically we have that problem, so we > > > can actually solve it, and so we can figure out why we failed to fix this > > > there earlier. > > > > Just bringing this up again - it seems we've not had much movement on > > this aside from the Wiki page. > > The wiki page currently still just recommends to set > "networkAccessManger->setAttribute(QNetworkRequest::FollowRedirectsAttribute, > true);" > > Which seems simple, but possible not what is enough in all cases. > > So my open questions here to be able to act on code I contribute to are: > > a) What about the mentioned QNetworkRequest::NoLessSafeRedirectPolicy, in > which cases should that be used and when not? For interacting with download.kde.org / files.kde.org, I would advise against using this policy, as they will in virtually all instances redirect to mirrors (who don't support https and are http only) > > b) What about the HSTS stuff, when is that recommended? That should be enabled yes. > > c) What is a sane number for QNetworkRequest::maximumRedirectsAllowed? 5 to 10 redirects is a relatively sane number I would expect. At the most I would expect our servers to issue a maximum of 3 redirects in a given chain of URLs. If it is longer than that then we are doing something wrong. > > Both in general and when it comes to KDE servers. > > Personally I am still unsure what the actual issue is. Why are redirects > needed at all. Why all the address changes all the time? The "U" in > "URL"/"URI" is for "uniform", not "unstable", isn't it ;) Please see my other email regarding this. > > Can you give some examples for URLs of resources our code uses on KDE servers, > and why they needed to change? Get Hot New Stuff functionality (Gen 1), originally using a static file tree under http://download.kde.org/khotnewstuff/ This needed to change for two reasons: 1) Mandatory HTTPS 2) The benefit of having these files mirrored, considering their extremely small size and declining client base (KDE 3 and parts of KDE 4) was negligible and creating more load on our systems to support the mirroring process than we got in terms of benefit of having them mirrored. We therefore transitioned to serving these through a CDN. Get Hot New Stuff functionality (Gen 2), originally using a dynamic web service at http://newstuff.kde.org/ and http://data.kstuff.org/ needed to change for two reasons: 1) Mandatory HTTPS 2) The dynamic web service had not been updated in several years, and was dependent on a very specific system setup we hadn't been able to replicate and needed to decomission due to it's age. We therefore needed to convert it to static files, and arrange for those to be hosted elsewhere in our systems. newstuff.kde.org now converts the requests sent to it to redirects to specific static files to keep applications using it working (which includes KF5 era applications who still actively use this and in at least one case continue to be released using this) Get Hot New Stuff functionality (Gen 3), originally used a file at http://download.kde.org/ocs/providers.xml (now at https://autoconfig.kde.org/ocs/providers.xml) This needed to change for two reasons: 1) Mandatory HTTPS 2) It was necessary for non-sysadmins (particularly those involved in running store.kde.org) to be able to update the file directly. As the server hosting download.kde.org is sensitive and doesn't support deploying changes from Git when they are committed, we had to move the file to a different subdomain which could support this. Marble maps, originally hosted under http://download.kde.org/ and later at http://files.kde.org/marble/maps/ and now at https://maps.kde.org/, This need to be moved for couple of reasons: 1) When we transitioned download.kde.org to be a mirror redirector, it was no longer possible for us to easily host non-mirrored resources under the same domain (and the maps weren't mirrored), requiring they be moved to files.kde.org (which as an added benefit also made it possible for developers to update the maps themselves) 2) Later, it was discovered that Marble performance for loading maps using files.kde.org after it transitioned to being a mirror redirector as well was quite poor due to the large number of http requests involved. We therefore shifted it to a CDN based resource which eliminated these performance issues, known as maps.kde.org. KStars resources, originally hosted under http://download.kde.org/apps/kstars/ needed to be moved to https://files.kde.org/ for the following reasons: 1) Mandatory HTTPS 2) T
Re: Banning QNetworkAccessManager
On Thu, Feb 20, 2020 at 9:57 PM Volker Krause wrote: > > On Wednesday, 19 February 2020 10:04:11 CET Ben Cooksley wrote: > > On Wed, Feb 19, 2020 at 9:30 PM Volker Krause wrote: > > > On Wednesday, 19 February 2020 08:05:01 CET Ben Cooksley wrote: > > > > On Mon, Feb 3, 2020 at 7:42 AM Volker Krause wrote: > > > > > I agree on the problem of QNAM's default, see also > > > > > https://conf.kde.org/en/ > > > > > akademy2019/public/events/135 on that subject. > > > > > > > > > > On Saturday, 1 February 2020 23:24:14 CET Ben Cooksley wrote: > > > > > [...] > > > > > > > > > > > Prior to now, i've taken the approach of advertising that > > > > > > QNetworkAccessManager is broken and needs a flag set within > > > > > > applications when used, however unfortunately this seems to have > > > > > > been > > > > > > an ineffective approach and cases of broken behaviour continue to > > > > > > appear (despite this brokeness being known about and advertised > > > > > > since > > > > > > back in the Qt 4.x series when this class was introduced) > > > > > > > > > > > > Therefore, given the Qt Project is unlikely to change their position > > > > > > > > > > > on this, I would like to propose the following: > > > > > The Qt Project is actually in favor of changing at least the > > > > > redirection > > > > > default to exactly what we want it to be. > > > > > https://codereview.qt-project.org/c/qt/qtbase/+/273175 has the change, > > > > > and > > > > > got approval both by Lars and one of the maintainers. It is merely > > > > > stuck > > > > > in dealing with the unit test fallout in Qt's CI that somebody has to > > > > > take care of. Doesn't immediately help us of course, but claiming Qt > > > > > is > > > > > unwilling to change this is simply wrong. > > > > > > > > > > > 1) That effective immediately, QNetworkAccessManager and it's > > > > > > children > > > > > > classes be banned and prohibited within KDE software, to be enforced > > > > > > by server side hooks. > > > > > > 2) That we fork QNetworkAccessManager and the associated classes > > > > > > within the appropriate Framework (to be determined), where the > > > > > > defective behaviour can then be corrected. > > > > > > > > > > While this might solve the redirection problem, it brings us yet > > > > > another > > > > > then unmaintained HTTP implementation next to the one in KIO, which is > > > > > a > > > > > far bigger problem long term. We need less of those, not more. > > > > > > > > > > To address the problem of people not using the correct defaults, I did > > > > > propose a much less extreme approach in the above mentioned talk at > > > > > Akademy, namely having a KNetworkAccessManager as a sub-class of QNAM > > > > > in > > > > > a low-tier frameworks that essentially just enables redirects and > > > > > HSTS. > > > > > QNAM's implementation isn't the problem after all, the defaults are. > > > > > > > > > > Commit hooks warning about QNAM usage might still be a good idea, but > > > > > as a > > > > > warning only. Hard blocking QNAM-using code would block at least three > > > > > repos I spend a lot of time on currently, none of which even talks to > > > > > KDE > > > > > servers. > > > > > > > > > > If you need to take more drastic measures against code not doing this > > > > > correctly regardless, you can do that my dropping the server-side > > > > > workarounds. Yes, that would break the affected application possibly, > > > > > but > > > > > at least it would not cause massive collateral damage for the people > > > > > using this correctly. > > > > > > > > > > It would also help to know where specifically we have that problem, so > > > > > we > > > > > can actually solve it, and so we can figure out why we failed to fix > > > > > this > > > > > there earlier. >
Re: Fwd: gsoc ktoblzcheck
On Tue, Feb 25, 2020 at 10:12 AM Victor Kukshiev wrote: > > I logged in, but Wiki don't allow access to Phabacker link. screenshot > https://imgur.com/4GropmI Hi Victor, I have investigated this and ascertained that this was caused by an issue with your Phabricator account (which unfortunately happens in some cases, and which we have never been able to trace the cause of) which prevents your email address from being marked as 'confirmed'. This in turn means your email address was never made available to the wiki - which is why it wouldn't allow you to message Ralf, as the Community Wiki did not have an email address for you. I have now corrected all of these issues for your account and synced your profile to the wiki, so you should now be able to access that page. Please note however that we prefer for communication to be made via a public channel for everything concerning GSoC, so I would recommend that you transition to a public mailing list for this as soon as is practical. Thanks, Ben Cooksley KDE Sysadmin > > пн, 24 февр. 2020 г. в 23:22, Carl Schwan : >> >> Hello, >> You might want to click on the contact Rhabacker link. If you are asked to >> login and don't have an account yet, you can create an account in >> identity.kde.org and then log through phabricator.kde.org. >> I hope it help. >> Best regards, >> Carl >> >> >> >> >> >> >> Original Message >> On 24 Feb 2020, 9:16 PM, Victor Kukshiev < andrey0bolkon...@gmail.com> wrote: >> >> >> >> Hello, I is a student of PetrSU (Petrozavodsk, Russia). I interested this >> idea and I want to participation. >> >> https://community.kde.org/GSoC/2020/Ideas#Project:_Provide_the_bank_data_needed_for_financial_applications_in_SQLite_format >> >> How to participate it? Who I must do?
Update on Status of Gitlab Migration
Good morning Community, I'm pleased to report that this week we reached a major milestone, with all the necessary technical components now being in place on our side for our migration to Gitlab to take place. This includes the replacement of the tooling that supports the anongit network, as well as implementation of a system to sync changes from Identity to Gitlab in real time. As part of this, the anongit network following the migration to Gitlab should pick up new repositories, along with changes to default branches, project descriptions, repository names and their path, and the deletion of repositories within 10 seconds of the change taking place on Gitlab (effectively making it real time) We are currently looking to confirm how repositories will be organised on Gitlab, after which we should be able to confirm how and when we perform the migration. Please note that only code hosting and code review will be transitioning at this time - CI and tasks will follow later on. Existing reviews on Phabricator will not be imported to Gitlab as part of this process and will need to be closed out on Phabricator. To help assist with this, it would be appreciated if all holders of a KDE Developer account could please login on our Gitlab instance (at https://invent.kde.org/) and ensure they are listed as a 'Developer' of the KDE group. You can do this by checking the 'Groups' tab of your Profile on Gitlab. Should you not be listed as a Developer of the group, and you have previously logged into Gitlab, please visit KDE Identity and make a change to your details there, which will trigger a sync of your account to Gitlab. Should anyone have any questions on the above, please let us know. Thanks, Ben Cooksley KDE Sysadmin
Notice of upcoming changes to the behaviour of the anongit network
Hi all, As part of the preparations for the move to Gitlab, and the rewrite of our anongit tooling, one of the things we have looked into is how the anongit network in general operates. As part of this, it has been observed that the git:// protocol is unencrypted, and thus vulnerable to intercept and manipulation by hostile actors. We have therefore decided that support for the git:// protocol to access KDE Git repositories will cease following our migration to Gitlab. Going forward, all anonymous access should take place instead over https, which is encrypted, and has the added benefit of offering support for redirects (should those be needed) Should anyone have any questions regarding this, please let us know. Thanks, Ben Cooksley KDE Sysadmin
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 10:46 PM Gleb Popov <6year...@gmail.com> wrote: > > > > On Sat, Apr 11, 2020 at 1:36 PM Ben Cooksley wrote: >> >> Good morning Community, >> >> I'm pleased to report that this week we reached a major milestone, >> with all the necessary technical components now being in place on our >> side for our migration to Gitlab to take place. >> >> This includes the replacement of the tooling that supports the anongit >> network, as well as implementation of a system to sync changes from >> Identity to Gitlab in real time. >> >> As part of this, the anongit network following the migration to Gitlab >> should pick up new repositories, along with changes to default >> branches, project descriptions, repository names and their path, and >> the deletion of repositories within 10 seconds of the change taking >> place on Gitlab (effectively making it real time) >> >> We are currently looking to confirm how repositories will be organised >> on Gitlab, after which we should be able to confirm how and when we >> perform the migration. >> >> Please note that only code hosting and code review will be >> transitioning at this time - CI and tasks will follow later on. >> Existing reviews on Phabricator will not be imported to Gitlab as part >> of this process and will need to be closed out on Phabricator. >> >> To help assist with this, it would be appreciated if all holders of a >> KDE Developer account could please login on our Gitlab instance (at >> https://invent.kde.org/) and ensure they are listed as a 'Developer' >> of the KDE group. You can do this by checking the 'Groups' tab of your >> Profile on Gitlab. >> >> Should you not be listed as a Developer of the group, and you have >> previously logged into Gitlab, please visit KDE Identity and make a >> change to your details there, which will trigger a sync of your >> account to Gitlab. > > > Hi Ben, thanks for working on this. Hi Gleb. > > My account https://invent.kde.org/users/arrowdodger/groups is not in > "Developers" group, and changing details on Identity website doesn't seem to > trigger any updates on Gitlab. > While on it, is it still impossible to turn "arrowdodger" into "arrowd"? The membership is of the 'KDE' group, with the access level of Developer - which is what I see on your account in this case. With regards to Developer usernames, that situation hasn't changed i'm afraid. Things are unlikely to change in this regard until Identity is replaced and we have the option of different usernames for different services. Cheers, Ben > >> >> Should anyone have any questions on the above, please let us know. >> >> Thanks, >> Ben Cooksley >> KDE Sysadmin
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:06 PM Simon Depiets wrote: > > Hello Ben, Hi Simon, > I don't see myself in the KDE group either (sdepiets) You needed to make a change on your Identity account to trigger the sync process. I've now done this and your membership has been automatically updated accordingly. Cheers, Ben > > Le sam. 11 avr. 2020 à 19:01, Ben Cooksley a écrit : >> >> On Sat, Apr 11, 2020 at 10:46 PM Gleb Popov <6year...@gmail.com> wrote: >> > >> > >> > >> > On Sat, Apr 11, 2020 at 1:36 PM Ben Cooksley wrote: >> >> >> >> Good morning Community, >> >> >> >> I'm pleased to report that this week we reached a major milestone, >> >> with all the necessary technical components now being in place on our >> >> side for our migration to Gitlab to take place. >> >> >> >> This includes the replacement of the tooling that supports the anongit >> >> network, as well as implementation of a system to sync changes from >> >> Identity to Gitlab in real time. >> >> >> >> As part of this, the anongit network following the migration to Gitlab >> >> should pick up new repositories, along with changes to default >> >> branches, project descriptions, repository names and their path, and >> >> the deletion of repositories within 10 seconds of the change taking >> >> place on Gitlab (effectively making it real time) >> >> >> >> We are currently looking to confirm how repositories will be organised >> >> on Gitlab, after which we should be able to confirm how and when we >> >> perform the migration. >> >> >> >> Please note that only code hosting and code review will be >> >> transitioning at this time - CI and tasks will follow later on. >> >> Existing reviews on Phabricator will not be imported to Gitlab as part >> >> of this process and will need to be closed out on Phabricator. >> >> >> >> To help assist with this, it would be appreciated if all holders of a >> >> KDE Developer account could please login on our Gitlab instance (at >> >> https://invent.kde.org/) and ensure they are listed as a 'Developer' >> >> of the KDE group. You can do this by checking the 'Groups' tab of your >> >> Profile on Gitlab. >> >> >> >> Should you not be listed as a Developer of the group, and you have >> >> previously logged into Gitlab, please visit KDE Identity and make a >> >> change to your details there, which will trigger a sync of your >> >> account to Gitlab. >> > >> > >> > Hi Ben, thanks for working on this. >> >> Hi Gleb. >> >> > >> > My account https://invent.kde.org/users/arrowdodger/groups is not in >> > "Developers" group, and changing details on Identity website doesn't seem >> > to trigger any updates on Gitlab. >> > While on it, is it still impossible to turn "arrowdodger" into "arrowd"? >> >> The membership is of the 'KDE' group, with the access level of >> Developer - which is what I see on your account in this case. >> >> With regards to Developer usernames, that situation hasn't changed i'm >> afraid. >> Things are unlikely to change in this regard until Identity is >> replaced and we have the option of different usernames for different >> services. >> >> Cheers, >> Ben >> >> > >> >> >> >> Should anyone have any questions on the above, please let us know. >> >> >> >> Thanks, >> >> Ben Cooksley >> >> KDE Sysadmin > > > > -- > Simon Depiets
Re: Notice of upcoming changes to the behaviour of the anongit network
On Sun, Apr 12, 2020 at 12:16 AM Ilya Bizyaev wrote: > > These gitconfig snippets will need an update after the transition: > > https://community.kde.org/Infrastructure/Git#Pushing > https://techbase.kde.org/Development/Git/Configuration#URL_Renaming Thanks for pointing these two pages out - i've now updated them to reflect this upcoming change. I have also removed mention of the repository tarballs which one of the pages was referring to, as that was a service we discontinued due to low usage many years ago. > > Best regards, > Ilya Cheers, Ben > > > Дата: Сб, 11 апр 2020 13:14:38 +0300 Ben Cooksley > написал(а) > > Hi all, > > As part of the preparations for the move to Gitlab, and the rewrite of > our anongit tooling, one of the things we have looked into is how the > anongit network in general operates. > > As part of this, it has been observed that the git:// protocol is > unencrypted, and thus vulnerable to intercept and manipulation by > hostile actors. > > We have therefore decided that support for the git:// protocol to > access KDE Git repositories will cease following our migration to > Gitlab. > > Going forward, all anonymous access should take place instead over > https, which is encrypted, and has the added benefit of offering > support for redirects (should those be needed) > > Should anyone have any questions regarding this, please let us know. > > Thanks, > Ben Cooksley > KDE Sysadmin > > >
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:31 PM Johan Ouwerkerk wrote: > > On Sat, Apr 11, 2020 at 11:36 AM Ben Cooksley wrote: > > > > Should anyone have any questions on the above, please let us know. > > > > Does the migration also mean that `git.kde.org` push URL will be > retired and would need to be remapped to `invent.kde.org`? Yes, the hostname git.kde.org will be fully retired as part of this transition. >From my understanding kdesrc-build will automatically pick this up once we update sysadmin/repo-metadata to show the new repository paths. This is something we'll confirm with mpyne though to ensure we can make the cutover as smooth as possible. Depending on how things look we may also make available a script that will update the configuration of a repository to reflect both the change in hostname as well as the change in path. > > In that case, it would be good to have a grace period after the > initial migration to Gitlab so kdesrc-build (etc.) could be updated > before the cut off date to perform this migration automatically for > the user. I expect such a grace period would not need to last very > long because the feature would be trivial to implement. > > Regards, > > - Johan Cheers, Ben
Re: Update on Status of Gitlab Migration
On Sun, Apr 12, 2020 at 11:04 AM Johan Ouwerkerk wrote: > > On Sun, Apr 12, 2020 at 12:49 AM Johan Ouwerkerk > wrote: > > > > Yes the only reason why a cleanup script might be needed is if the > > logical path used to express the repo in dependency information > > changes at the same time. E.g. suppose a `frameworks/kf5foo` gets > > remapped to `frameworks/kf5/foo` or something like that. In that case > > unless you use the flat repository layout, kdesrc-build would try to > > clone a new `frameworks/kf5/foo` repo, leaving your old > > `frameworks/kf5foo` to consume some wasted disk space. > > > > This is obviously a poor example, but the same problem occurs if > something moves from playground to extragear. Basically if the > `projectpath` YAML key changes. I had been considering adding the Gitlab Project ID number to the YAML metadata files as a way of allowing us to track projects through their whole lifetime. > > Regards, > > - Johan Cheers, Ben
Downtime for SVN - Action Required
Hi all, As you may be aware, KDE is currently undertaking a migration to Gitlab. As part of this, we also need to migrate our Subversion repository from the server it currently resides on to the new system that also hosts Gitlab. To simplify our systems we will also transition management of SSH keys to Gitlab when we do this. It is intended at this time that we perform the migration of the Subversion repository next weekend. We therefore need all holders of a Developer account, even if they only use our Subversion repository, to login to Gitlab and upload their SSH keys there. Our Gitlab instance can be found at https://invent.kde.org/ and you should be able to login there using your KDE Identity (identity.kde.org) details. Additionally, following this migration we will begin limiting access to the Subversion repository to those actively using it. The list of people whose access will be continued can be found at https://cgit.kde.org/repo-management.git/tree/svn-ssh-keys/users-list Should you still need access to the repository, but are not on the above list, please let us know by emailing sysad...@kde.org. If anyone has any questions on the above, please let us know. Regards, Ben Cooksley KDE Sysadmin
Re: Downtime for SVN - Action Required
On Sun, Apr 12, 2020 at 10:12 PM Albert Astals Cid wrote: > > El diumenge, 12 d’abril de 2020, a les 10:56:12 CEST, Ben Cooksley va > escriure: > > Additionally, following this migration we will begin limiting access > > to the Subversion repository to those actively using it. The list of > > people whose access will be continued can be found at > > https://cgit.kde.org/repo-management.git/tree/svn-ssh-keys/users-list > > Can you explain what's the rationale for what seems a rather arbitrary > decision that may even go against our manifesto? This was already gone over as part of the thread with the subject "Sysadmin Load Reduction: Subversion Infrastructure", which you participated in. It is far from an arbitrary decision, and has already been discussed. In addition to what was discussed then, while implementing the retrieval of SSH keys from Gitlab it has been discovered that there is no way to easily retrieve all the SSH keys belonging to the members of a given group in bulk from it's API. The only way to do this by fetching the keys for each user who needs access in turn which creates a scalability problem. As Subversion access is only needed by those working on translations, I don't see the problem here, and anyone with a developer account who believes they may need access is welcome to request access - keeping in mind that they should only be doing so if it is something they need. > > Do people not on that list at least get a nice message saying "please request > access here" when trying to commit to SVN? > Their SSH key will not be recognised and they will be refused access, so it will not be possible to provide any form of message. > Cheers, > Albert > Regards, Ben
Re: Notice of upcoming changes to the behaviour of the anongit network
On Mon, Apr 13, 2020 at 2:24 PM Ian Wadham wrote: > > Hi Ben, Hi Ian, > > > On 11 Apr 2020, at 8:14 pm, Ben Cooksley wrote: > > > > Hi all, > > > > As part of the preparations for the move to Gitlab, and the rewrite of > > our anongit tooling, one of the things we have looked into is how the > > anongit network in general operates. > > > > As part of this, it has been observed that the git:// protocol is > > unencrypted, and thus vulnerable to intercept and manipulation by > > hostile actors. > > > > We have therefore decided that support for the git:// protocol to > > access KDE Git repositories will cease following our migration to > > Gitlab. > > > > Going forward, all anonymous access should take place instead over > > https, which is encrypted, and has the added benefit of offering > > support for redirects (should those be needed) > > > > Should anyone have any questions regarding this, please let us know. > > I am a former KDE developer but AFAIK my account and access have long since > expired. > > However I still hover on kde-devel, kde-games-devel and and Apple OSX list > (MacPorts users) and am sometimes able to suggest solutions to problems that > come up, for which I need read-only access to source code from anongit. I > have some questions regarding how things will work in future. > > 1. Will the general public still have open access to browse KDE source code > repositories on screens? Not sure what you are referring to by 'screens' here sorry. > > 2. Will I be able to clone a read-only copy of a KDE repository (I have no > intention of committing to central repos)? > > 3. Will I be able to do the above without having to know any account ID, > password or encryption key? In terms of open access, they will continue to be browsable through the Gitlab web interface (at https://invent.kde.org/) and can be cloned over https (either from the anongit network, or from Gitlab itself) without any authentication or account being required. It is only the git:// protocol that is being discontinued. > > Thanks and best regards, > Ian Wadham. > Cheers, Ben
Re: Update on Status of Gitlab Migration
On Tue, Apr 14, 2020 at 3:35 AM Nate Graham wrote: > > On 4/13/20 4:44 AM, Albert Vaca Cintora wrote: > > Regarding this: is the subdomain going to stay invent.kde.org once we > > have officially moved? I find it's a bit confusing to use that instead > > of gitlab.kde.org > > I agree. gitlab.kde.org would make more sense to me, mirroring > phabricator.kde.org. > There is no intention to change the name from invent.kde.org. We have a long precedent of not using the name of the software for the name in DNS, and Gitlab is continuing with this, for example: - bugs.kde.org is run by Bugzilla - dot.kde.org is run by Drupal - techbase.kde.org is run by Mediawiki - conf.kde.org is run by Frab - reimbursements.kde.org is run by travel-support-program - websvn.kde.org is run by ViewVC - build.kde.org and binary-factory.kde.org are both run by Jenkins - stats.kde.org is run by Matomo (formerly Piwik) - survey.kde.org is run by Limesurvey For essentially all of the above, calling it after the software running it makes no sense, and given that in some cases we have multiple instances would have made things more confusing (jenkins1.kde.org and jenkins2.kde.org anyone?) Phabricator and Reviewboard were the only ones to not follow this rule, and that was an error on our part. Given that there is a popular instance at Gitlab.com, referring to ours as "KDE Invent" is more likely to ensure newcomers are not confused (as they may not be aware that you can setup an instance of Gitlab on your own systems) Regards, Ben
Re: Update on Status of Gitlab Migration
On Tue, 14 Apr 2020, 11:45 am Aleix Pol, wrote: > On Mon, Apr 13, 2020 at 8:30 PM Ben Cooksley wrote: > > > > On Tue, Apr 14, 2020 at 3:35 AM Nate Graham wrote: > > > > > > On 4/13/20 4:44 AM, Albert Vaca Cintora wrote: > > > > Regarding this: is the subdomain going to stay invent.kde.org once > we > > > > have officially moved? I find it's a bit confusing to use that > instead > > > > of gitlab.kde.org > > > > > > I agree. gitlab.kde.org would make more sense to me, mirroring > > > phabricator.kde.org. > > > > > > > There is no intention to change the name from invent.kde.org. > > > > We have a long precedent of not using the name of the software for the > > name in DNS, and Gitlab is continuing with this, for example: > > - bugs.kde.org is run by Bugzilla > > - dot.kde.org is run by Drupal > > - techbase.kde.org is run by Mediawiki > > - conf.kde.org is run by Frab > > - reimbursements.kde.org is run by travel-support-program > > - websvn.kde.org is run by ViewVC > > - build.kde.org and binary-factory.kde.org are both run by Jenkins > > - stats.kde.org is run by Matomo (formerly Piwik) > > - survey.kde.org is run by Limesurvey > > > > For essentially all of the above, calling it after the software > > running it makes no sense, and given that in some cases we have > > multiple instances would have made things more confusing > > (jenkins1.kde.org and jenkins2.kde.org anyone?) > > > > Phabricator and Reviewboard were the only ones to not follow this > > rule, and that was an error on our part. > > > > Given that there is a popular instance at Gitlab.com, referring to > > ours as "KDE Invent" is more likely to ensure newcomers are not > > confused (as they may not be aware that you can setup an instance of > > Gitlab on your own systems) > > > > Regards, > > Ben > > We also use git.kde.org and svn.kde.org. > The name git.kde.org will be retired as part of the move to Gitlab. The SSH host keys for the two machines are different, so it would cause more issues than it doesn't to keep the hostname active. We've also never supported HTTP access to the master copy of the repositories so there isn't anything to keep alive there either. > It would too mimic what others are doing at gitlab.gnome.org and > gitlab.freedesktop.org. So at the very least we'll want a redirect. > Why do we need to mimic them? If you Google "KDE Gitlab" then the first hit is invent.kde.org. > Aleix > Cheers, Ben >
Re: Update on Status of Gitlab Migration
On Mon, Apr 13, 2020 at 9:29 AM Johan Ouwerkerk wrote: > > On Sun, Apr 12, 2020 at 12:49 AM Johan Ouwerkerk > wrote: > > > > > > > > We may need to do on-the-fly conversion of the kde: repo paths if they > > > won't be expressible as 'kde:foo' in the future, but we should have the > > > information needed to do this in kdesrc-build to make this happen > > > on-the-fly. > > > > > > > Yes, this should be fairly straight forward: we could do a `git remote > > set-url` based on what the repo metadata tells us before updating a > > local clone. In fact: we could build this right now and sell it as > > "automagically recover your upstream". :) > > > > I might try to hack something up tomorrow or monday for that. > > > > A basic version of this is now available via: > https://invent.kde.org/kde/kdesrc-build/-/merge_requests/27 > With this feature sysadmin should now be free to change the repopath > value in the metadata YAML and kdesrc-build will reconfigure the > remote URL appropriately automatically. This works as long as the same > `kde:$path` expression works for both fetch and push, i.e. the layout > on the anongit.kde.org network must match with the layout of > git.kde.org/invent.kde.org. If necessary a simple path prefix change > could still work with a minor update to the pushInsteadOf mapping, > e.g. when repos on invent are mapped to kde/ instead of > /. > > You can also experiment with setting invent.kde.org as the push URL by > setting x-invent-kde-push-urls to true in your rc file. The effect > should be visible through git remote -vv afterwards. (Disable the > setting and re-run again afterwards because this will obviously break > your push URLs as long as the Gitlab migration hasn't completed yet). Many thanks for sending this patch through Johan, it's appreciated. As one of the options we are looking at involves grouping repositories, having the ability for kdesrc-build to seamlessly handle this transition is definitely appreciated. > > Regards, > > - Johan Cheers, Ben
Re: Update on Status of Gitlab Migration
On Tue, Apr 14, 2020 at 2:37 PM Nate Graham wrote: > > On 4/13/20 6:59 PM, Ben Cooksley wrote: > > Why do we need to mimic them? > > > > If you Google "KDE Gitlab" then the first hit is invent.kde.org > > <http://invent.kde.org>. > > To flip it around: why do we need to do something different? I don't > think it's about mimicking anyone else, but rather using the most > intuitively obvious domain name rather than some arbitrarily different one. > > No matter what, we're all going to be talking about "KDE GitLab" or > "KDE's GitLab instance". The title of the relevant documentation page is > "GitLab" (much as the Phabricator documentation page was named > "Phabricator"). And when the migration is complete, we're all going to > announce that "KDE is now using GitLab!" So the cat's out of the bag on > using the word "GitLab" and avoiding some number of people looking for > our stuff at gitlab.com and not finding it there. invent.kde.org doesn't > avoid that at all. > > Given that, which is more confusing: explaining to people that we have a > GitLab instance at invent.kde.org, or explaining to people that we have > a GitLab instance at gitlab.kde.org? There is no difference to be honest, the amount of explaining is exactly the same. We're also not alone in giving ours a different name - Debian did as well (theirs is at salsa.debian.org). They didn't see fit to setup a compatibility "gitlab.debian.org" hostname either. > > I know that over the next few years I'm going to be directing hundreds > of people towards our infrastructure, and I would rather be able to say > "please submit a pull request at gitlab.kde.org" rather than "please > submit a pull request at KDE's GitLab instance at invent.kde.org." Sorry, but the "KDE's Gitlab instance" in your second sentence is superfluous. You could just as easily say "please submit a pull request on invent.kde.org", not sure why you need to explicitly mention it is Gitlab. If people have already found the source code, chances are they will have already found invent.kde.org anyway - as they will have probably found it via a web browser, and Gitlab will be the only web interface to our repositories (CGit is being discontinued) > > I know this probably feels like annoyingly extreme bikeshedding, but, I > dunno, names are important. > > Nate Cheers, Ben
Re: Notice of upcoming changes to the behaviour of the anongit network
On Wed, Apr 15, 2020 at 12:49 AM Myriam Schweingruber wrote: > > > > On Sat, 11 Apr 2020 at 20:29, Ben Cooksley wrote: >> >> On Sun, Apr 12, 2020 at 12:16 AM Ilya Bizyaev wrote: >> > >> > These gitconfig snippets will need an update after the transition: >> > >> > https://community.kde.org/Infrastructure/Git#Pushing >> > https://techbase.kde.org/Development/Git/Configuration#URL_Renaming >> >> Thanks for pointing these two pages out - i've now updated them to >> reflect this upcoming change. > > > Another wiki page that needs updating I guess: > > https://community.kde.org/Get_Involved/development Thanks for pointing this out Myriam, looks like someone has updated this page. > > Regards, Myriam Cheers, Ben > > > -- > Proud member of the Amarok and KDE Community > Protect your freedom and support the work of the FSFE: > http://www.fsfe.org > Please don't send me proprietary file formats, > use ISO standard ODF instead (ISO/IEC 26300)
Re: Proposal: "Noteworthy" label for Promo application updates
On Thu, Apr 16, 2020 at 2:05 AM Nate Graham wrote: > > I support a standardized tag/keyword/whatever to attract promo attention > to something. However if we use a GitLab tag, then we won't be capturing > information from Phabricator patches during the time before we've fully > transitioned to GitLab. Should we instead use a new NOTEWORTHY: Bugzilla > keyword (like BUG: or FEATURE:) or wait do do this until after the > GitLab migration is complete? The Gitlab migration (for code hosting and review) is a matter of weeks away at this stage, so the window of time you're thinking of there is very small. > > Nate > Cheers, Ben > > > On 4/15/20 6:19 AM, Julian wrote: > > Hello everyone, > > > > As you all are probably aware, KDE Promo writes monthly updates on what > > happens within our applications and also promotes the releases done by > > the Release Service (like Dolphin 20.04, Kate 20.04, Kontact, etc.). For > > these updates, we need to get all important new features and bug fixes > > that are present in the latest releases. > > > > Up until now, we have reached out to developers every time we wanted to > > compile such feature lists. This approach has several drawbacks: > > > > - Projects occasionally forget to add their new features. This leaves us > > with the choice to either seek out all changes ourselves (which is very > > time consuming) or omit the application in the writeup (which is not > > good either)- Smaller projects might be left out because we do not know > > that they have released something cool- There is an inevitable waiting > > period between us asking for a feature list and the completion of the > > list. This can lead to delays in the announcement process and lowers the > > quality of the announcement (for example, because translators don't have > > the time to translate the text before it is made public) > > > > We would therefore like to propose a different approach. > > > > 1. Get a standardized "Noteworthy" label added on Invent. 2. Whenever > > a noteworthy change is merged (that is, a cool new feature or an > > important bug fix), the reviewers add said label. 3. When Promo drafts > > release notes or other updates, we can filter for said label and > > instantly see all relevant changes. > > > > This ensures that no important contributions are overlooked and results > > in a much smoother workflow for Promo. We'd like to get feedback from > > developers on this proposal and if no major objections are raised, > > implement this workflow change as soon as code hosting is fully > > transitioned to GitLab. > > > > Best regards,Julian / xyquadrat on behalf of KDE Promo > > > > P.S.: Please note that we still need your notes on changes for the 20.04 > > release, however. Visit https://share.kde.org/s/SJqzmemQ3PsrmGR and add > > in noteworthy features and bug fixes there. > > >
Re: Disable shortcuts in GitLab invent.kde.org
On Thu, Apr 16, 2020 at 6:41 AM David Hurka wrote: > Hi David, > According to this help page, it should be possible to disable shortcuts in > gitlab: > https://docs.gitlab.com/ee/user/shortcuts.html > But the said option is not available in our GitLab instance. Is there any way > to disable shortcuts? I've just checked, and the option is definitely available to me on our instance. You have tried pressing ? when on any screen in our Gitlab yes? If you have, i'd be inclined to suspect that Falkon is interfering in the proper operation of Gitlab. > > I really want to disable them, because they don’t have any advantage over > Falkons spatial navigation, and do a lot of harm. It happens so often to me > that I unfocus the text boxes for some reason, and then trigger various of > these shortcuts instead. :( > > Cheers, Ben
Re: Proposal: "Noteworthy" label for Promo application updates
On Sun, Apr 19, 2020 at 8:34 PM XYQuadrat wrote: > > If we implement such a system (that is, just resurrect what is already > present), I definitely think it would be sensible to make guidelines on > what commits we are interested in publicly available. > The thing I'm not sure about is how easy it'd be to automate grabbing > all the CHANGELOG commits from a given date range - can someone more > experienced with our git/repo infrastructure shed some light here? That depends on whether you have local, up to date, clones of the repositories in question. If you have them locally it should be reasonably trivial with a bit of scripting to get a list of commits and the repositories to which they belong. If you don't, then the only other option you would have would be Gitlab's APIs but that isn't ideal from an infrastructural point of view. > > Best regards, > Julian / xyquadrat > Cheers, Ben > On 17.04.20 01:19, Johannes Zarl-Zierl wrote: > > On Donnerstag, 16. April 2020 23:15:18 CEST Nate Graham wrote: > >> On 4/16/20 2:38 PM, Albert Astals Cid wrote: > >>> It may make sense to highlight them a bit more somehow, suggestions > >>> welcome i guess. > >> The promo people just need a list of all CHANGELOG entries in a release > >> so they can rewrite it in more human-friendly terms. Right now this is > >> done manually by me and others by looking through commit logs and blog > >> posts and adding the equivalent of CHANGELOG text into an > >> etherpad/share.kde.org document. Automating that somehow would be nice. > > From a developer point of view, a tag in the commit itself seems like > > a nice > > interface. One thing I fear, though, is that people like me forget to > > actually > > add it to the commit before pushing, so maybe something that can be added > > later would definitely have its advantages. > > > > Personally, I'd like to have some clear guidelines from the promo team > > on what > > kind of features they are looking for. That way it's way easier for me to > > match those expectations ;-) > > > > Btw. isn't the DIGEST keyword as documented in the standard commit > > template > > basically the same idea? > > > > Cheers, > > Johannes > > > > > >
