Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:18 AM David Faure  wrote:

> On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> > On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > > For frameworks, I think we should be able to write a one-time script
> that
> > > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > > metadata (and the platforms listed in metainfo.yaml) ?
> >
> > Yes, that should work nicely (although that information now lives in
> > sysadmin/repo-metadata, dependencies folder)
>
> All done for Frameworks.
>
> The script that collects platforms from metainfo.yaml won't be useful for
> other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> qt5 is attached, in case it's useful to anyone else.
>

Thanks for sorting this David, very much appreciated.


> > Once we've transitioned across to the .kde-ci.yml files the existing
> > dependency metadata and branch group files will no longer be required.
> > (We'll need to work out a way of exporting this information for easy
> > consumption by kdesrc-build and other clients before it can be removed
> > though)
>
> OK. Duplication is bad so I agree :)
>

Cheers,
Ben


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:46 AM Nicolas Fella  wrote:

> On 05.09.21 08:13, Ben Cooksley wrote:
> > Hi all,
> >
> > This morning after much work i'm happy to announce that the new
> > generation CI scripts intended for use with Gitlab CI successfully
> > completed their first build (of ECM, and then subsequently of
> > KCoreAddons).
> >
> > This begins our first steps towards transferring CI over from Jenkins
> > to Gitlab.
> >
> > These scripts are 'Gitlab native' and are designed to work in an
> > environment where they will be running on merge requests, tags as well
> > as all branches of our repositories - something the existing scripts
> > were completely incompatible with. While there are still some
> > infrastructural elements to put in place (such as 'seed jobs' to mass
> > rebuild all projects after substantial changes and 'cleanup jobs' to
> > remove old builds from the Package Registry) the core elements needed
> > for initial testing of these scripts are now ready.
> >
> > For those curious, the results of those initials runs can be found at
> > https://invent.kde.org/groups/teams/ci-artifacts/-/packages
> > 
> >
> > Due to the challenges associated with having to handle all of the
> > different forms of build and the versioning of this information, these
> > scripts also represent a radical change in how CI builds will be
> > conducted - with a large part of the configuration of the jobs
> > themselves, including information on project dependencies now shifting
> > to files located within the repository themselves. Those who monitor
> > commits to Frameworks repositories will have noticed the appearance of
> > these '.kde-ci.yml' files in some repositories.
> >
> > To assist in the future rollout of the CI system it would be
> > appreciated if projects could please begin setting up these files
> > within their respective repositories.
> > You can find a template detailing the available options at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> >
> >
> > Where possible please avoid overriding the system defaults except
> > where needed by your project (ie. don't just copy the template in full)
> > The defaults mirror the template and can be found at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> >
> >
> > In terms of the format of the 'Dependencies' section, please bear in
> > mind the following:
> > - For the 'on' section, the special value '@all' can be used to mean
> > "All platforms" to avoid needing to update the file in the event
> > additional platforms are added in the future
> > - For the 'require' section:
> >   1) Please only include the projects you *explicitly* depend on.
> > Dependencies of your dependencies will be included automatically
> >   2) When specifying a project, you must use the repository path on
> > Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
> > are no longer in use and will not work.
> >   3) There are three special values for the branch name - '@same',
> > '@latest' and '@stable' which can be used to refer to the branch/tag
> > of a dependency.
> >   '@same' means the branch name should be the same as that of the
> > project being built and should be used when declaring dependencies
> > among projects in a release group.
> >   '@latest' and '@stable' refer to the corresponding branches
> > defined in the branch-rules.yml file, which can be found in
> > sysadmin/repo-metadata
> >
> > As a general rule, unless you require the absolute latest version of
> > another project in another release unit, it is recommended that you
> > use @stable.
> > Please avoid using explicit branch names unless absolutely necessary.
> >
> > At this time it is expected that the first set of Gitlab CI builds
> > using the new scripts may be conducted for Frameworks within the next
> > week or so, assuming the build of the seed jobs goes according to plan.
> >
> > Once the scripts have been proven successfully for Frameworks, we will
> > look at extending them to projects that depend only on Frameworks and
> > repositories within their release unit and finally will extend them to
> > projects with more complex dependency requirements. It is expected
> > that the switch will be flipped on the Frameworks builds sometime in
> > the next week or two.
> >
> > Please let me know if you have any questions on the above.
> >
> > Thanks,
> > Ben Cooksley
> > KDE Sysadmin
>
> Hi Ben,
>

Hi Nicolas,


> thanks for your work on this!
>
> How would I best encode a dependency like "On Linux and FreeBSD depend
> on kwayland, but not on Windows and macOS"?
>

I would suggest doing something like the followin

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 11:03 AM Michael Reeves  wrote:

> How do we get a visual on exactly which lines are covered by auto testing
> and which aren't?
>

Please see
https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
for more details on how this works on Merge Requests.
To my knowledge this isn't exposed outside of Merge Requests.

Cheers,
Ben

Re: Gitlab CI - Inbound

[Michael Reeves]

How do we get a visual on exactly which lines are covered by auto testing
and which aren't?

Re: Gitlab CI - Inbound

[Nicolas Fella]

On 05.09.21 08:13, Ben Cooksley wrote:

Hi all,

This morning after much work i'm happy to announce that the new
generation CI scripts intended for use with Gitlab CI successfully
completed their first build (of ECM, and then subsequently of
KCoreAddons).

This begins our first steps towards transferring CI over from Jenkins
to Gitlab.

These scripts are 'Gitlab native' and are designed to work in an
environment where they will be running on merge requests, tags as well
as all branches of our repositories - something the existing scripts
were completely incompatible with. While there are still some
infrastructural elements to put in place (such as 'seed jobs' to mass
rebuild all projects after substantial changes and 'cleanup jobs' to
remove old builds from the Package Registry) the core elements needed
for initial testing of these scripts are now ready.

For those curious, the results of those initials runs can be found at
https://invent.kde.org/groups/teams/ci-artifacts/-/packages


Due to the challenges associated with having to handle all of the
different forms of build and the versioning of this information, these
scripts also represent a radical change in how CI builds will be
conducted - with a large part of the configuration of the jobs
themselves, including information on project dependencies now shifting
to files located within the repository themselves. Those who monitor
commits to Frameworks repositories will have noticed the appearance of
these '.kde-ci.yml' files in some repositories.

To assist in the future rollout of the CI system it would be
appreciated if projects could please begin setting up these files
within their respective repositories.
You can find a template detailing the available options at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml


Where possible please avoid overriding the system defaults except
where needed by your project (ie. don't just copy the template in full)
The defaults mirror the template and can be found at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml


In terms of the format of the 'Dependencies' section, please bear in
mind the following:
- For the 'on' section, the special value '@all' can be used to mean
"All platforms" to avoid needing to update the file in the event
additional platforms are added in the future
- For the 'require' section:
  1) Please only include the projects you *explicitly* depend on.
Dependencies of your dependencies will be included automatically
  2) When specifying a project, you must use the repository path on
Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
are no longer in use and will not work.
  3) There are three special values for the branch name - '@same',
'@latest' and '@stable' which can be used to refer to the branch/tag
of a dependency.
  '@same' means the branch name should be the same as that of the
project being built and should be used when declaring dependencies
among projects in a release group.
  '@latest' and '@stable' refer to the corresponding branches
defined in the branch-rules.yml file, which can be found in
sysadmin/repo-metadata

As a general rule, unless you require the absolute latest version of
another project in another release unit, it is recommended that you
use @stable.
Please avoid using explicit branch names unless absolutely necessary.

At this time it is expected that the first set of Gitlab CI builds
using the new scripts may be conducted for Frameworks within the next
week or so, assuming the build of the seed jobs goes according to plan.

Once the scripts have been proven successfully for Frameworks, we will
look at extending them to projects that depend only on Frameworks and
repositories within their release unit and finally will extend them to
projects with more complex dependency requirements. It is expected
that the switch will be flipped on the Frameworks builds sometime in
the next week or two.

Please let me know if you have any questions on the above.

Thanks,
Ben Cooksley
KDE Sysadmin


Hi Ben,

thanks for your work on this!

How would I best encode a dependency like "On Linux and FreeBSD depend
on kwayland, but not on Windows and macOS"?

Cheers

Nico

Re: Gitlab CI - Inbound

[David Faure]

On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > For frameworks, I think we should be able to write a one-time script that
> > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > metadata (and the platforms listed in metainfo.yaml) ?
> 
> Yes, that should work nicely (although that information now lives in
> sysadmin/repo-metadata, dependencies folder)

All done for Frameworks.

The script that collects platforms from metainfo.yaml won't be useful for 
other KDE modules, but the script that collects deps from dependency-data-kf5-
qt5 is attached, in case it's useful to anyone else.

> Once we've transitioned across to the .kde-ci.yml files the existing
> dependency metadata and branch group files will no longer be required.
> (We'll need to work out a way of exporting this information for easy
> consumption by kdesrc-build and other clients before it can be removed
> though)

OK. Duplication is bad so I agree :)

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5


collect_deps.pl
Description: Perl program

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:

> On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> > It would be appreciated if people could please work on getting these
> files
> > populated in Frameworks (as everyone needs those) as well as in their own
> > repositories as they are required before we can proceed much further.
>
> Like this
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?
>

Yes - just two small tweaks needed for that one but it is otherwise good to
ship.


> For frameworks, I think we should be able to write a one-time script that
> generates .kde-ci.yml files using the dependencies listed in kde-build-
> metadata (and the platforms listed in metainfo.yaml) ?
>

Yes, that should work nicely (although that information now lives in
sysadmin/repo-metadata, dependencies folder)

Once we've transitioned across to the .kde-ci.yml files the existing
dependency metadata and branch group files will no longer be required.
(We'll need to work out a way of exporting this information for easy
consumption by kdesrc-build and other clients before it can be removed
though)


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
Thanks,
Ben

Re: Gitlab CI - Inbound

[David Faure]

On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> It would be appreciated if people could please work on getting these files
> populated in Frameworks (as everyone needs those) as well as in their own
> repositories as they are required before we can proceed much further.

Like this https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?

For frameworks, I think we should be able to write a one-time script that 
generates .kde-ci.yml files using the dependencies listed in kde-build-
metadata (and the platforms listed in metainfo.yaml) ?

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:

> Hi all,
>

Hi all,


> This morning after much work i'm happy to announce that the new generation
> CI scripts intended for use with Gitlab CI successfully completed their
> first build (of ECM, and then subsequently of KCoreAddons).
>
> This begins our first steps towards transferring CI over from Jenkins to
> Gitlab.
>
> These scripts are 'Gitlab native' and are designed to work in an
> environment where they will be running on merge requests, tags as well as
> all branches of our repositories - something the existing scripts were
> completely incompatible with. While there are still some infrastructural
> elements to put in place (such as 'seed jobs' to mass rebuild all projects
> after substantial changes and 'cleanup jobs' to remove old builds from the
> Package Registry) the core elements needed for initial testing of these
> scripts are now ready.
>

As an update, an initial version of the seed job tooling is now ready,
however testing that tooling requires that dependency information be
available.
This requires that the .kde-ci.yml files be populated in repositories.

It would be appreciated if people could please work on getting these files
populated in Frameworks (as everyone needs those) as well as in their own
repositories as they are required before we can proceed much further.


> For those curious, the results of those initials runs can be found at
> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>
> Due to the challenges associated with having to handle all of the
> different forms of build and the versioning of this information, these
> scripts also represent a radical change in how CI builds will be conducted
> - with a large part of the configuration of the jobs themselves, including
> information on project dependencies now shifting to files located within
> the repository themselves. Those who monitor commits to Frameworks
> repositories will have noticed the appearance of these '.kde-ci.yml' files
> in some repositories.
>
> To assist in the future rollout of the CI system it would be appreciated
> if projects could please begin setting up these files within their
> respective repositories.
> You can find a template detailing the available options at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>
> Where possible please avoid overriding the system defaults except where
> needed by your project (ie. don't just copy the template in full)
> The defaults mirror the template and can be found at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>
> In terms of the format of the 'Dependencies' section, please bear in mind
> the following:
> - For the 'on' section, the special value '@all' can be used to mean "All
> platforms" to avoid needing to update the file in the event additional
> platforms are added in the future
> - For the 'require' section:
>   1) Please only include the projects you *explicitly* depend on.
> Dependencies of your dependencies will be included automatically
>   2) When specifying a project, you must use the repository path on
> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
> longer in use and will not work.
>   3) There are three special values for the branch name - '@same',
> '@latest' and '@stable' which can be used to refer to the branch/tag of a
> dependency.
>   '@same' means the branch name should be the same as that of the
> project being built and should be used when declaring dependencies among
> projects in a release group.
>   '@latest' and '@stable' refer to the corresponding branches defined
> in the branch-rules.yml file, which can be found in sysadmin/repo-metadata
>
> As a general rule, unless you require the absolute latest version of
> another project in another release unit, it is recommended that you use
> @stable.
> Please avoid using explicit branch names unless absolutely necessary.
>
> At this time it is expected that the first set of Gitlab CI builds using
> the new scripts may be conducted for Frameworks within the next week or so,
> assuming the build of the seed jobs goes according to plan.
>
> Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories within their release unit and finally will extend them to
> projects with more complex dependency requirements. It is expected that the
> switch will be flipped on the Frameworks builds sometime in the next week
> or two.
>
> Please let me know if you have any questions on the above.
>
> Thanks,
> Ben Cooksley
> KDE Sysadmin
>

Thanks,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 9:04 PM Gilles Caulier 
wrote:

> Congratulations to Ben for this important step in the KDE CI workflow.
>
> I'm sure that it's too far ahead, but we will be interested later, when
> the infrastructure is ready in gitlab, to run all gitlab CI with digiKam
> code. Of course this requires solving a lot of dependencies, but please
> keep in mind in the future. This can also be a project to delegate to a
> student later : "port digiKam CI to Gitlab"...
>

The existing Docker images will be carried over, so all non-KDE
dependencies will continue to be available from Day 1.

The important part here is that projects start getting their .kde-ci.yml
files in place.
This is especially the case if you are a project that other projects depend
on (as is the case for libraries, as well as a handful of applications)


> My best regards
>
> Gilles Caulier
>

Regards,
Ben


> Gilles Caulier
>
> Le dim. 5 sept. 2021 à 10:38, Halla Rempt  a écrit :
>
>> On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
>> > Hi all,
>> >
>> > This morning after much work i'm happy to announce that the new
>> generation
>> > CI scripts intended for use with Gitlab CI successfully completed their
>> > first build (of ECM, and then subsequently of KCoreAddons).
>> >
>>
>> Yay!
>>
>> Halla
>>
>>
>>
>>

Re: Gitlab CI - Inbound

[Gilles Caulier]

Congratulations to Ben for this important step in the KDE CI workflow.

I'm sure that it's too far ahead, but we will be interested later, when the
infrastructure is ready in gitlab, to run all gitlab CI with digiKam code.
Of course this requires solving a lot of dependencies, but please keep in
mind in the future. This can also be a project to delegate to a student
later : "port digiKam CI to Gitlab"...

My best regards

Gilles Caulier

Gilles Caulier

Le dim. 5 sept. 2021 à 10:38, Halla Rempt  a écrit :

> On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> > Hi all,
> >
> > This morning after much work i'm happy to announce that the new
> generation
> > CI scripts intended for use with Gitlab CI successfully completed their
> > first build (of ECM, and then subsequently of KCoreAddons).
> >
>
> Yay!
>
> Halla
>
>
>
>

Re: Gitlab CI - Inbound

[Halla Rempt]

On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> Hi all,
> 
> This morning after much work i'm happy to announce that the new generation
> CI scripts intended for use with Gitlab CI successfully completed their
> first build (of ECM, and then subsequently of KCoreAddons).
> 

Yay!

Halla