Blacklisting of PIM from the CI system
Hi all, This morning I went to look into provisioning a new Windows Builder node for the CI system, but hit a roadblock created by PIM currently failing to build from source. As some background to this, we use Craft to provide various dependencies of our projects that aren't provided by Windows itself (such as Qt, gettext, poppler and so forth). From time to time this is updated to provide newer versions of software, but unless it is necessary for a project, the CI nodes themselves often aren't updated. We use the same approach for both Linux and FreeBSD, except those dependencies are provided by the distribution for those two. This means that when it comes time to provision a new node, all of the nodes need to be updated. As these changes essentially always break compatibility in some form or another this makes it necessary for us to rebuild all the KDE software as well. It is therefore not possible to proceed with any of the above while something is failing to build. Which is where the problem with PIM comes in - because it currently has many repositories failing to build from source on all platforms those builds are enabled for (including Linux and FreeBSD). Given that the PIM project mailing list is emailed by the CI system, and that the changes in one case were pushed over 2 days ago, there is no excuse for this series of build failures. In addition to all of the above, this round of updates was to lay the ground work for adding additional dependencies which are necessary for the builds of Digikam and SubtitleComposer to commence on the CI system for Windows. These failures by PIM have therefore indirectly harmed other KDE projects. As this is not the first time that PIM has caused issues in this manner, I would therefore like to proceed with blacklisting PIM from the CI system. This would include prohibiting other projects from depending on any part of PIM, so those projects that have a required dependency on PIM would also have their builds removed by this. Whilst I would prefer another solution to this, given that it is a recurring issue that makes maintenance of the CI system substantially harder, I see the removal of PIM from the equation as the only reasonable path forward. Should the PIM developers wish to avoid this consequence for their actions, they will need to provide an action plan as to how this will be avoided in the future. Fixing the current set of failures will not prevent this blacklisting action from being carried out - as a recurring issue it needs a permanent solution. Regards, Ben Cooksley KDE Sysadmin
Re: Blacklisting of PIM from the CI system
sigh... On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: [...] > Which is where the problem with PIM comes in - because it currently > has many repositories failing to build from source on all platforms > those builds are enabled for (including Linux and FreeBSD). looking at this right now, at least three errors seem transient (ie. a manual rebuild "fixed" them), one I can't explain yet is this one: https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/job/ kf5-qt5%20SUSEQt5.12/39/console This is about using new API in KF 5.65, but the use is guarded by a corresponding version check #ifdef. Which KF5 is the CI building against, latest release, latest master, or maybe something in between (the latter would explain this)? I'll look through the rest as time permits. > Given that the PIM project mailing list is emailed by the CI system, > and that the changes in one case were pushed over 2 days ago, there is > no excuse for this series of build failures. I try to actively look at the CI state every 24h, however that sometimes fails when I'm on the road or otherwise busy. I do see the email notifications, but given the high number of transient failures (usually due to unfortunately timed dependency bumps), they are of limited use for me for raising an alarm in cases of actual breakage. > In addition to all of the above, this round of updates was to lay the > ground work for adding additional dependencies which are necessary for > the builds of Digikam and SubtitleComposer to commence on the CI > system for Windows. These failures by PIM have therefore indirectly > harmed other KDE projects. > > As this is not the first time that PIM has caused issues in this > manner, I would therefore like to proceed with blacklisting PIM from > the CI system. This would include prohibiting other projects from > depending on any part of PIM, so those projects that have a required > dependency on PIM would also have their builds removed by this. Of course stuff tends to break more often when you look at 50+ actively developed repos compared to a single barely alive one. And yes, I do very well understand that this can be frustrating. > Whilst I would prefer another solution to this, given that it is a > recurring issue that makes maintenance of the CI system substantially > harder, I see the removal of PIM from the equation as the only > reasonable path forward. > > Should the PIM developers wish to avoid this consequence for their > actions, they will need to provide an action plan as to how this will > be avoided in the future. I cannot realistically promise more than what I am already doing on this (as outlined above), the combination of me not able to pay enough attention to the CI state and things blowing up in multiple repos on the same day is very rare though. If that's not enough, others need to help here as well. > Fixing the current set of failures will not prevent this blacklisting > action from being carried out - as a recurring issue it needs a > permanent solution. What do you envision this permanent solution to look like? Regards, Volker signature.asc Description: This is a digitally signed message part.
Re: Blacklisting of PIM from the CI system
On Sun, Dec 1, 2019 at 10:17 AM Volker Krause wrote: > > sigh... > > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: > [...] > > Which is where the problem with PIM comes in - because it currently > > has many repositories failing to build from source on all platforms > > those builds are enabled for (including Linux and FreeBSD). > > looking at this right now, at least three errors seem transient (ie. a manual > rebuild "fixed" them), one I can't explain yet is this one: > > https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/job/ > kf5-qt5%20SUSEQt5.12/39/console > > This is about using new API in KF 5.65, but the use is guarded by a > corresponding version check #ifdef. Which KF5 is the CI building against, > latest release, latest master, or maybe something in between (the latter would > explain this)? > The CI system essentially uses the latest master of Frameworks - with the slight catch of it the build being up to a week out of date. The Frameworks build is updated by the "Dependency Build" jobs that run for each Product/Branch/Platform combination once a week. The last time the Dependency Build job ran for the Applications kf5-qt5 SUSEQt5.12 combination was about 2 days ago. https://build.kde.org/job/Administration/job/Dependency%20Build%20Applications%20kf5-qt5%20SUSEQt5.12/40/console Unfortunately this run failed due to breakage in 'pimcommon'. Before it failed, it did get the chance to build Sonnet so that is up to date as of the date of that build. > I'll look through the rest as time permits. Thanks, that would be appreciated. > > > Given that the PIM project mailing list is emailed by the CI system, > > and that the changes in one case were pushed over 2 days ago, there is > > no excuse for this series of build failures. > > I try to actively look at the CI state every 24h, however that sometimes fails > when I'm on the road or otherwise busy. I do see the email notifications, but Thanks for doing so. > given the high number of transient failures (usually due to unfortunately > timed dependency bumps), they are of limited use for me for raising an alarm > in cases of actual breakage. *nod* With regards to the dependency bumps, the best way to avoid these is to first bump version numbers, and then increase the dependency requirements in the projects in a second round of commits, pushed separately, once the version bump builds have finished. This is from my understanding how David does it for Frameworks and it works flawlessly. This solution has been proposed in the past, but was rejected by the PIM team who insisted that it was the CI system that should instead sequence the builds correctly (something which is unscalable to implement, and from my understanding impossible with Gitlab CI) > > > In addition to all of the above, this round of updates was to lay the > > ground work for adding additional dependencies which are necessary for > > the builds of Digikam and SubtitleComposer to commence on the CI > > system for Windows. These failures by PIM have therefore indirectly > > harmed other KDE projects. > > > > As this is not the first time that PIM has caused issues in this > > manner, I would therefore like to proceed with blacklisting PIM from > > the CI system. This would include prohibiting other projects from > > depending on any part of PIM, so those projects that have a required > > dependency on PIM would also have their builds removed by this. > > Of course stuff tends to break more often when you look at 50+ actively > developed repos compared to a single barely alive one. And yes, I do very well > understand that this can be frustrating. > > > Whilst I would prefer another solution to this, given that it is a > > recurring issue that makes maintenance of the CI system substantially > > harder, I see the removal of PIM from the equation as the only > > reasonable path forward. > > > > Should the PIM developers wish to avoid this consequence for their > > actions, they will need to provide an action plan as to how this will > > be avoided in the future. > > I cannot realistically promise more than what I am already doing on this (as > outlined above), the combination of me not able to pay enough attention to the > CI state and things blowing up in multiple repos on the same day is very rare > though. If that's not enough, others need to help here as well. > > > Fixing the current set of failures will not prevent this blacklisting > > action from being carried out - as a recurring issue it needs a > > permanent solution. > > What do you envision this permanent solution to look like? It's hard to say. Given the current problem, which is changes being actively made that break the build, the ideal solution would be the developers who are making these breakages to actively keep an eye on their jobs on the CI system. To avoid the issue with 'transient' failures due to new functions, etc being introduced to libraries then being used immedia
Re: Blacklisting of PIM from the CI system
On Sunday, 1 December 2019 04:00:19 CET Ben Cooksley wrote: > On Sun, Dec 1, 2019 at 10:17 AM Volker Krause wrote: > > sigh... > > > > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: > > [...] > > > > > Which is where the problem with PIM comes in - because it currently > > > has many repositories failing to build from source on all platforms > > > those builds are enabled for (including Linux and FreeBSD). > > > > looking at this right now, at least three errors seem transient (ie. a > > manual rebuild "fixed" them), one I can't explain yet is this one: > > > > https://build.kde.org/job/Applications/view/Everything/job/kpimtextedit/jo > > b/ kf5-qt5%20SUSEQt5.12/39/console > > > > This is about using new API in KF 5.65, but the use is guarded by a > > corresponding version check #ifdef. Which KF5 is the CI building against, > > latest release, latest master, or maybe something in between (the latter > > would explain this)? > > The CI system essentially uses the latest master of Frameworks - with > the slight catch of it the build being up to a week out of date. > The Frameworks build is updated by the "Dependency Build" jobs that > run for each Product/Branch/Platform combination once a week. > > The last time the Dependency Build job ran for the Applications > kf5-qt5 SUSEQt5.12 combination was about 2 days ago. > https://build.kde.org/job/Administration/job/Dependency%20Build%20Applicatio > ns%20kf5-qt5%20SUSEQt5.12/40/console > > Unfortunately this run failed due to breakage in 'pimcommon'. > Before it failed, it did get the chance to build Sonnet so that is up > to date as of the date of that build. I suspect that's where the problem is. The "missing" commit from Sonnet is from Nov 26, the dependency build is from Nov 28, yet the kpimtextedit build still "sees" the old version. The same problem also exists with kwidgetaddons and korganizer/knotes. I looked in the code of those cases, and tested this locally. This builds against 5.64 and latest 5.65, when I switch frameworks to something in between I can reproduce the issues we are seeing on the CI here too. Not sure how to fix that in the code, I can't version-check for this... > > > Given that the PIM project mailing list is emailed by the CI system, > > > and that the changes in one case were pushed over 2 days ago, there is > > > no excuse for this series of build failures. > > > > I try to actively look at the CI state every 24h, however that sometimes > > fails when I'm on the road or otherwise busy. I do see the email > > notifications, but > Thanks for doing so. > > > given the high number of transient failures (usually due to unfortunately > > timed dependency bumps), they are of limited use for me for raising an > > alarm in cases of actual breakage. > > *nod* > > With regards to the dependency bumps, the best way to avoid these is > to first bump version numbers, and then increase the dependency > requirements in the projects in a second round of commits, pushed > separately, once the version bump builds have finished. This is from > my understanding how David does it for Frameworks and it works > flawlessly. > > This solution has been proposed in the past, but was rejected by the > PIM team who insisted that it was the CI system that should instead > sequence the builds correctly (something which is unscalable to > implement, and from my understanding impossible with Gitlab CI) I agree with that solution, and it's how I try to do dependency bumps when necessary. (I'll reply to the ideas how to improve the situation going forward separately/later, I'm in a rush now, sorry). Regards, Volker > > > In addition to all of the above, this round of updates was to lay the > > > ground work for adding additional dependencies which are necessary for > > > the builds of Digikam and SubtitleComposer to commence on the CI > > > system for Windows. These failures by PIM have therefore indirectly > > > harmed other KDE projects. > > > > > > As this is not the first time that PIM has caused issues in this > > > manner, I would therefore like to proceed with blacklisting PIM from > > > the CI system. This would include prohibiting other projects from > > > depending on any part of PIM, so those projects that have a required > > > dependency on PIM would also have their builds removed by this. > > > > Of course stuff tends to break more often when you look at 50+ actively > > developed repos compared to a single barely alive one. And yes, I do very > > well understand that this can be frustrating. > > > > > Whilst I would prefer another solution to this, given that it is a > > > recurring issue that makes maintenance of the CI system substantially > > > harder, I see the removal of PIM from the equation as the only > > > reasonable path forward. > > > > > > Should the PIM developers wish to avoid this consequence for their > > > actions, they will need to provide an action plan as to how this will > > > be avoi
Re: Blacklisting of PIM from the CI system
On Sunday, 1 December 2019 04:00:19 CET Ben Cooksley wrote: > On Sun, Dec 1, 2019 at 10:17 AM Volker Krause wrote: > > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: [...] > > > Fixing the current set of failures will not prevent this blacklisting > > > action from being carried out - as a recurring issue it needs a > > > permanent solution. > > > > What do you envision this permanent solution to look like? > > It's hard to say. > > Given the current problem, which is changes being actively made that > break the build, the ideal solution would be the developers who are > making these breakages to actively keep an eye on their jobs on the CI > system. That isn't an entirely fair statement IMHO, the changes triggering this were perfectly fine with either the latest Frameworks release or a recent enough master, just not the delayed master we had available on the CI at this point. Seeing this hit other master-tracking builds too (e.g. OpenSuse in #kontact this morning), I completely agree though this needs a better solution overall. Tracking the latest release would be one approach, but from direct discussion I understand that's currently not a viable solution due to Plasma's needs. Not allowing changes for master compatibility (with the corresponding version #ifdefs) is also counter-productive though, as it prevents early testing of Frameworks changes (even if just locally). So the best I can think of is making sure this situation is widely enough understood, and we have a way to find out which exact revisions of Frameworks are deployed. Then we can maybe get to the point where we can defer such commits until a recent enough revision is available, which IIUC would usually take 48-72h. > To avoid the issue with 'transient' failures due to new functions, etc > being introduced to libraries then being used immediately in other > projects, i'd suggest that the developers introduce a delay (30 > minutes should be sufficient) between the pushes to give the system a > chance to build the updated libraries. Agreed, for many people that's already standard practice I think. > In the long run it would be > nice to shrink the size of the PIM dependency graph, either through > consolidating repositories or reorganising the code to cut the number > of dependencies, which would reduce the number of times you'd need to > wait. The complexity of the dependency graph is also a problem for onboarding new people, and with kdelibs4support gone IMHO the largest technical dept here. It's being worked on, albeit slowly, currently we are losing ~1 module per release I think. Regards, Volker signature.asc Description: This is a digitally signed message part.
Re: Blacklisting of PIM from the CI system
On Tuesday, December 3, 2019 2:52:31 PM EST Volker Krause wrote: > The complexity of the dependency graph is also a problem for onboarding new > people, and with kdelibs4support gone IMHO the largest technical dept here. > It's being worked on, albeit slowly, currently we are losing ~1 module per > release I think. > Silly questions Why do we need 50-60 repos to build kontact? why not 1 repo for kontact and all its stuff? (I miss kdepim) do Krita or any of the other large applications have so many repositories devoted to them? After all these years since the svn->git move I still understand why so many repostitories for Kontact. For frameworks it makes sense, but I don't get it for applications
Re: Blacklisting of PIM from the CI system
On Tuesday, 3 December 2019 22:13:43 CET Allen Winter wrote: > On Tuesday, December 3, 2019 2:52:31 PM EST Volker Krause wrote: > > The complexity of the dependency graph is also a problem for onboarding > > new > > people, and with kdelibs4support gone IMHO the largest technical dept > > here. > > It's being worked on, albeit slowly, currently we are losing ~1 module per > > release I think. > > Silly questions > > Why do we need 50-60 repos to build kontact? > why not 1 repo for kontact and all its stuff? (I miss kdepim) > > do Krita or any of the other large applications have so many repositories > devoted to them? > > After all these years since the svn->git move I still understand why so many > repostitories for Kontact. For frameworks it makes sense, but I don't get > it for applications 50 repos for this is too much IMHO, I agree. There's reasons for not having this in a single one though, one being that various parts are used externally as well (former kdepimlibs basically). This is what we try to address with moving things to Frameworks. There's also things like Akonadi or Kleo being used externally though, that's much harder to Frameworkify. And then there are a few obsolete reasons, like the refactoring done for Kontact Mobile 10 years ago, or for Kube. Another aspect to look at is whether those repos have a well defined scope. Good examples are IMHO eventviews/incidenceeditor, bad ones are libkdepim/ kdepim-apps-libs/pimcommon/etc. The former hurt much less than the latter I think. The strategy discussed at the PIM meetings for this is two-fold: continue with Frameworkification of the well-scoped and externally used modules, and try to dissolve the libraries with undefined scope by moving stuff to more appropriate places. Especially the latter isn't exactly a fun job, therefore this is progressing very slowly. Going to a mono repo setup would make building easier for PIM contributors, but it would not solve the problem for new people to find their way through the various libraries with unclear scope, so IMHO this is a problem we have to address regardless of the repo layout. Regards, Volker signature.asc Description: This is a digitally signed message part.
Re: Blacklisting of PIM from the CI system
On Tue, Dec 3, 2019 at 8:55 PM Volker Krause wrote: > > On Sunday, 1 December 2019 04:00:19 CET Ben Cooksley wrote: > > On Sun, Dec 1, 2019 at 10:17 AM Volker Krause wrote: > > > On Saturday, 30 November 2019 19:14:38 CET Ben Cooksley wrote: > [...] > > > > Fixing the current set of failures will not prevent this blacklisting > > > > action from being carried out - as a recurring issue it needs a > > > > permanent solution. > > > > > > What do you envision this permanent solution to look like? > > > > It's hard to say. > > > > Given the current problem, which is changes being actively made that > > break the build, the ideal solution would be the developers who are > > making these breakages to actively keep an eye on their jobs on the CI > > system. > > That isn't an entirely fair statement IMHO, the changes triggering this were > perfectly fine with either the latest Frameworks release or a recent enough > master, just not the delayed master we had available on the CI at this point. > Seeing this hit other master-tracking builds too (e.g. OpenSuse in #kontact > this morning), I completely agree though this needs a better solution overall. > > Tracking the latest release would be one approach, but from direct discussion > I understand that's currently not a viable solution due to Plasma's needs. Not > allowing changes for master compatibility (with the corresponding version > #ifdefs) is also counter-productive though, as it prevents early testing of > Frameworks changes (even if just locally). So the best I can think of is > making sure this situation is widely enough understood, and we have a way to > find out which exact revisions of Frameworks are deployed. Then we can maybe > get to the point where we can defer such commits until a recent enough > revision is available, which IIUC would usually take 48-72h. Random thought to make dependency problems more obvious: glue the git sha into the cmake package version for frameworks (when built from git) so it finds "5.65.0.abcd123" given us a lead on what is actually available. https://phabricator.kde.org/D24641 would give us the foundation for that. Not sure how viable this is, or even if it adds much value. After all everyone still needs to know to check the specific in the build logs. Food for thought I guess. HS
