Re: Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
On sabato 6 maggio 2017 11:37:51 CEST, Ben Cooksley wrote: Hi everyone, This is going to be quite a long email, my apologies in advance for that. If you use the CI system regularly as part of your development flow it is important you read this email in it's entirety as your action will probably be required. If you are aware of a list I have missed in the above, please feel free to forward it there. As some will have been aware of for some time we currently have a problem where changing the system base is an incredibly disruptive activity. We also have issues where builds sometimes fail due to files disappearing mid-build or installations being inconsistent, and excessive numbers of emails are generated. To top this off, everyone has to use the same underlying "base" system for performing builds which sets up conflicts between projects. Oh, and we also only perform CI for Linux. To solve these problems we've been working on a Next Generation CI system which introduces a new concept called 'Products'. In short, a Product is a release unit, such as Frameworks, Plasma, KDE Applications, which groups a series of repositories which are developed and subsequently released together. A preview of this system can be viewed at https://build-sandbox.kde.org/ Products as part of their definition are able to define a set of 'Platforms' on which they build. At the moment we have three Platforms available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. Adding additional Platforms to this mix is fairly easy, as long as the code can be built there. Qt will now be considered as part of the base system, and is something we will no longer build ourselves. Each Product / Platform definition is fully independent. This will allow for different products to build against different versions of Qt among other things for instance. This is the first part that requires your action: we'd like developers, particularly those whose development relies on bleeding edge system software to assist in creating and maintaining (Linux) Platform images. If you're interested in this, please let me know. As part of the shift to the Products paradigm, a number of changes have been made to how projects are built and dependencies managed. This particularly affects dependencies on projects which are outside your Product (Frameworks is outside of Plasma and Applications for instance). These dependencies will now be satisfied through "Dependency Build" jobs, which will be executed once a week. As a result the master version of Frameworks will no longer be available outside Frameworks itself. This change is one of the big ones which massively reduces the effort required to change base systems and thus hugely improves the maintainability of the CI system. It is therefore quite important and necessary, and also isolates the rest of the CI system from breakages lower down in the stack which have happened in the past. This is the second point that requires your attention. If your development process is dependent on using the latest development version of something which is located in another product, then we will need to add that to your Product. If this affects you, please start a new thread (CC'ing sysadmin and kde-core-devel along with your Product's main list) stating which specific repositories you need and providing one to two lines of explaination for each. Note that requests for the entirety of Frameworks won't be accepted - each must be requested on an individual basis with an explanation given for why your development process is dependent on the master version of that Framework. With the change to Products as a core concept for driving the CI system, this does leave Extragear somewhat in a bit of an unusual situation. At the moment we're planning to deal with this by having Extragear be a 'Product' with all of them combined together. If anyone in Extragear has any comments to make on this they're certainly welcome here. Which brings me to the third point of attention. We'll only be adding projects to the Next Gen CI system at their request going forward. For Frameworks, Applications and Plasma this is something which we're essentially assuming we're going to receive from their release managers, so we'll take care of defining the initial Products for those. For Extragear projects, please respond to this thread if you'd like CI coverage (to continue) to be provided to you. What's the role of kde-build-metadata.git in the new CI? If an extragear project is already defined there, won't it be automatically built by the new CI? Thanks for all who have reached this point - my apologies again for it's length. Note that the Next Gen system isn't finished yet - Notifications are yet to be setup and there are a few other touches left to be made. Regards, Ben Cooksley KDE Sysadmin Cheers, Elvis
Re: Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
On Sat, May 6, 2017 at 10:15 PM, Elvis Angelaccio wrote: > On sabato 6 maggio 2017 11:37:51 CEST, Ben Cooksley wrote: >> >> Hi everyone, >> >> This is going to be quite a long email, my apologies in advance for that. >> If you use the CI system regularly as part of your development flow it >> is important you read this email in it's entirety as your action will >> probably be required. If you are aware of a list I have missed in the >> above, please feel free to forward it there. >> >> As some will have been aware of for some time we currently have a >> problem where changing the system base is an incredibly disruptive >> activity. We also have issues where builds sometimes fail due to files >> disappearing mid-build or installations being inconsistent, and >> excessive numbers of emails are generated. To top this off, everyone >> has to use the same underlying "base" system for performing builds >> which sets up conflicts between projects. Oh, and we also only perform >> CI for Linux. >> >> To solve these problems we've been working on a Next Generation CI >> system which introduces a new concept called 'Products'. In short, a >> Product is a release unit, such as Frameworks, Plasma, KDE >> Applications, which groups a series of repositories which are >> developed and subsequently released together. >> >> A preview of this system can be viewed at https://build-sandbox.kde.org/ >> >> Products as part of their definition are able to define a set of >> 'Platforms' on which they build. At the moment we have three Platforms >> available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. >> Adding additional Platforms to this mix is fairly easy, as long as the >> code can be built there. Qt will now be considered as part of the base >> system, and is something we will no longer build ourselves. >> >> Each Product / Platform definition is fully independent. This will >> allow for different products to build against different versions of Qt >> among other things for instance. >> >> This is the first part that requires your action: we'd like >> developers, particularly those whose development relies on bleeding >> edge system software to assist in creating and maintaining (Linux) >> Platform images. If you're interested in this, please let me know. >> >> As part of the shift to the Products paradigm, a number of changes >> have been made to how projects are built and dependencies managed. >> This particularly affects dependencies on projects which are outside >> your Product (Frameworks is outside of Plasma and Applications for >> instance). These dependencies will now be satisfied through >> "Dependency Build" jobs, which will be executed once a week. As a >> result the master version of Frameworks will no longer be available >> outside Frameworks itself. >> >> This change is one of the big ones which massively reduces the effort >> required to change base systems and thus hugely improves the >> maintainability of the CI system. It is therefore quite important and >> necessary, and also isolates the rest of the CI system from breakages >> lower down in the stack which have happened in the past. >> >> This is the second point that requires your attention. If your >> development process is dependent on using the latest development >> version of something which is located in another product, then we will >> need to add that to your Product. If this affects you, please start a >> new thread (CC'ing sysadmin and kde-core-devel along with your >> Product's main list) stating which specific repositories you need and >> providing one to two lines of explaination for each. >> >> Note that requests for the entirety of Frameworks won't be accepted - >> each must be requested on an individual basis with an explanation >> given for why your development process is dependent on the master >> version of that Framework. >> >> With the change to Products as a core concept for driving the CI >> system, this does leave Extragear somewhat in a bit of an unusual >> situation. At the moment we're planning to deal with this by having >> Extragear be a 'Product' with all of them combined together. If anyone >> in Extragear has any comments to make on this they're certainly >> welcome here. >> >> Which brings me to the third point of attention. We'll only be adding >> projects to the Next Gen CI system at their request going forward. For >> Frameworks, Applications and Plasma this is something which we're >> essentially assuming we're going to receive from their release >> managers, so we'll take care of defining the initial Products for >> those. For Extragear projects, please respond to this thread if you'd >> like CI coverage (to continue) to be provided to you. > > > What's the role of kde-build-metadata.git in the new CI? If an extragear > project is already defined there, won't it be automatically built by the new > CI? kde-build-metadata continues to be consulted for: a) Dependencies b) Branch Group -> Branch associations
Re: Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
On Saturday May 06 2017 22:01:58 Ben Cooksley wrote: >On Sat, May 6, 2017 at 9:58 PM, René J.V. Bertin wrote: >> On Saturday May 06 2017 21:37:51 Ben Cooksley wrote: >> >>>'Platforms' on which they build. At the moment we have three Platforms >>>available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. >>>Adding additional Platforms to this mix is fairly easy, as long as the >>>code can be built there. Qt will now be considered as part of the base >>>system, and is something we will no longer build ourselves. >> >> One remark about this: Qt 5.7 is not the most issue-free version but I >> understand why the 5.6 LTS version was not preferred instead. However, there >> is 1 thing with using stock Qt that's potentially problematic on a CI. It >> lacks a QtDBus patch that Qt cannot at the moment incorporate but that >> everyone should be using because it prevents crashing on exit under certain >> conditions. > >It'll be up to the system distributor in that case to include the >patch. We shouldn't be including patches the system distributor isn't >in any case, as that will lead to a different environment compared to >our users. That works both ways. It's Qt's intention that distributions and "home builders" include the patch, so you can also adopt the position that a CI intended to find issues with KDE software shouldn't be vulnerable to issues in KDE dependencies that are not supposed to be present on user systems. Note that I myself wouldn't really know which position to adopt... but I would probably avoid ambiguity and provide the dependencies with all known issues either explicitly fixed or explicitly NOT fixed (the latter being the easy solution). Depending on the decisions of a single "system distributor" chosen among the myriad of distributors (for Linux at least) seems ... problematic in this aspect. >Note that for Windows and Mac we shouldn't be doing D-Bus anyway as >they're alien to that platform This is not exactly true and IMHO not a decision to be imposed by something like a CI. The DBus daemon includes support for both platforms and should be seen as a cross-platform desktop IPC solution which is also the only IPC interface Qt provides to my knowledge. Interdicting DBus use on Mac and MS Windows would make those platforms inaccessible to projects like KDEPIM. I don't think they'd appreciate that. In fact, any application using KWallet will probably run into issues. >(and the CI Tooling won't launch D-Bus >as part of it's test environment setup there as a result) The QtDBus issue itself does *not* depend on whether a DBus daemon is running or not, it's purely internal to Qt. R.
Re: Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
On Sat, May 6, 2017 at 9:58 PM, René J.V. Bertin wrote: > On Saturday May 06 2017 21:37:51 Ben Cooksley wrote: > >>'Platforms' on which they build. At the moment we have three Platforms >>available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. >>Adding additional Platforms to this mix is fairly easy, as long as the >>code can be built there. Qt will now be considered as part of the base >>system, and is something we will no longer build ourselves. > > One remark about this: Qt 5.7 is not the most issue-free version but I > understand why the 5.6 LTS version was not preferred instead. However, there > is 1 thing with using stock Qt that's potentially problematic on a CI. It > lacks a QtDBus patch that Qt cannot at the moment incorporate but that > everyone should be using because it prevents crashing on exit under certain > conditions. It'll be up to the system distributor in that case to include the patch. We shouldn't be including patches the system distributor isn't in any case, as that will lead to a different environment compared to our users. Note that for Windows and Mac we shouldn't be doing D-Bus anyway as they're alien to that platform (and the CI Tooling won't launch D-Bus as part of it's test environment setup there as a result) > > R. Cheers, Ben
Re: Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
On Saturday May 06 2017 21:37:51 Ben Cooksley wrote: >'Platforms' on which they build. At the moment we have three Platforms >available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. >Adding additional Platforms to this mix is fairly easy, as long as the >code can be built there. Qt will now be considered as part of the base >system, and is something we will no longer build ourselves. One remark about this: Qt 5.7 is not the most issue-free version but I understand why the 5.6 LTS version was not preferred instead. However, there is 1 thing with using stock Qt that's potentially problematic on a CI. It lacks a QtDBus patch that Qt cannot at the moment incorporate but that everyone should be using because it prevents crashing on exit under certain conditions. R.
Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes
Hi everyone, This is going to be quite a long email, my apologies in advance for that. If you use the CI system regularly as part of your development flow it is important you read this email in it's entirety as your action will probably be required. If you are aware of a list I have missed in the above, please feel free to forward it there. As some will have been aware of for some time we currently have a problem where changing the system base is an incredibly disruptive activity. We also have issues where builds sometimes fail due to files disappearing mid-build or installations being inconsistent, and excessive numbers of emails are generated. To top this off, everyone has to use the same underlying "base" system for performing builds which sets up conflicts between projects. Oh, and we also only perform CI for Linux. To solve these problems we've been working on a Next Generation CI system which introduces a new concept called 'Products'. In short, a Product is a release unit, such as Frameworks, Plasma, KDE Applications, which groups a series of repositories which are developed and subsequently released together. A preview of this system can be viewed at https://build-sandbox.kde.org/ Products as part of their definition are able to define a set of 'Platforms' on which they build. At the moment we have three Platforms available: Ubuntu Xenial Qt 5.7, Windows Qt 5.7 and FreeBSD Qt 5.7. Adding additional Platforms to this mix is fairly easy, as long as the code can be built there. Qt will now be considered as part of the base system, and is something we will no longer build ourselves. Each Product / Platform definition is fully independent. This will allow for different products to build against different versions of Qt among other things for instance. This is the first part that requires your action: we'd like developers, particularly those whose development relies on bleeding edge system software to assist in creating and maintaining (Linux) Platform images. If you're interested in this, please let me know. As part of the shift to the Products paradigm, a number of changes have been made to how projects are built and dependencies managed. This particularly affects dependencies on projects which are outside your Product (Frameworks is outside of Plasma and Applications for instance). These dependencies will now be satisfied through "Dependency Build" jobs, which will be executed once a week. As a result the master version of Frameworks will no longer be available outside Frameworks itself. This change is one of the big ones which massively reduces the effort required to change base systems and thus hugely improves the maintainability of the CI system. It is therefore quite important and necessary, and also isolates the rest of the CI system from breakages lower down in the stack which have happened in the past. This is the second point that requires your attention. If your development process is dependent on using the latest development version of something which is located in another product, then we will need to add that to your Product. If this affects you, please start a new thread (CC'ing sysadmin and kde-core-devel along with your Product's main list) stating which specific repositories you need and providing one to two lines of explaination for each. Note that requests for the entirety of Frameworks won't be accepted - each must be requested on an individual basis with an explanation given for why your development process is dependent on the master version of that Framework. With the change to Products as a core concept for driving the CI system, this does leave Extragear somewhat in a bit of an unusual situation. At the moment we're planning to deal with this by having Extragear be a 'Product' with all of them combined together. If anyone in Extragear has any comments to make on this they're certainly welcome here. Which brings me to the third point of attention. We'll only be adding projects to the Next Gen CI system at their request going forward. For Frameworks, Applications and Plasma this is something which we're essentially assuming we're going to receive from their release managers, so we'll take care of defining the initial Products for those. For Extragear projects, please respond to this thread if you'd like CI coverage (to continue) to be provided to you. Thanks for all who have reached this point - my apologies again for it's length. Note that the Next Gen system isn't finished yet - Notifications are yet to be setup and there are a few other touches left to be made. Regards, Ben Cooksley KDE Sysadmin
