Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 10:28 PM Vlad Zahorodnii 
wrote:

> On 9/7/21 1:21 PM, Ben Cooksley wrote:
> > On Tue, Sep 7, 2021 at 10:14 PM Vlad Zahorodnii  > > wrote:
> >
> > On 9/7/21 12:22 PM, Ben Cooksley wrote:
> >  > On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii
> > mailto:vlad.zahorod...@kde.org>
> >  >  > >> wrote:
> >  >
> >  > On 9/5/21 3:18 PM, David Faure wrote:
> >  >  > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley
> wrote:
> >  >  >> On Sun, Sep 5, 2021 at 10:22 PM David Faure
> > mailto:fa...@kde.org>
> >  > >> wrote:
> >  >  >>> For frameworks, I think we should be able to write a
> > one-time
> >  > script that
> >  >  >>> generates .kde-ci.yml files using the dependencies
> listed in
> >  > kde-build-
> >  >  >>> metadata (and the platforms listed in metainfo.yaml) ?
> >  >  >>
> >  >  >> Yes, that should work nicely (although that information
> > now lives in
> >  >  >> sysadmin/repo-metadata, dependencies folder)
> >  >  >
> >  >  > All done for Frameworks.
> >  >  >
> >  >  > The script that collects platforms from metainfo.yaml
> won't be
> >  > useful for
> >  >  > other KDE modules, but the script that collects deps from
> >  > dependency-data-kf5-
> >  >  > qt5 is attached, in case it's useful to anyone else.
> >  >
> >  > Is there a file that maps legacy project paths to gitlab
> > project paths?
> >  > dependency-data-kf5-qt5 lists projects with their legacy
> paths.
> >  >
> >  >
> >  > The individual project YAML files in sysadmin/repo-metadata
> > contain this
> >  > information.
> >  > The legacy project path can be found under 'projectpath' while the
> >  > Gitlab paths are under 'repopath'
> >
> > Thanks! I've attached a quick and dirty python script that parses
> > project dependencies from repo-metadata and prints corresponding
> gitlab
> > project paths.
> >
> > Example usage
> >
> > python project-deps.py --repo-metadata
> > /data/projects/src/repo-metadata/ kde/workspace/kwin
> >
> > Output
> >
> > Dependencies:
> > 'requires':
> >   'frameworks/extra-cmake-modules': '@stable'
> >   'plasma/kdecoration': '@stable'
> >   'plasma/kscreenlocker': '@stable'
> >   'plasma/kwayland-integration': '@stable'
> >   'plasma/breeze': '@stable'
> >   'plasma/kwayland-server': '@stable'
> >
> >
> > Please note that the above is missing a 'on' section as required for
> > each Dependencies section in the .kde-ci.yml file.
> Yeah... Is there a database or something that can be queried to fill in
> the "on" section?
>

I'm afraid this is information we have not really collected elsewhere at
this time.

Valid values are: Linux, FreeBSD, Windows, macOS, Android
(capitalisation sensitive)

Cheers,
Ben

Re: Gitlab CI - Inbound

[Vlad Zahorodnii]

On 9/7/21 1:21 PM, Ben Cooksley wrote:
On Tue, Sep 7, 2021 at 10:14 PM Vlad Zahorodnii > wrote:


On 9/7/21 12:22 PM, Ben Cooksley wrote:
 > On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii
mailto:vlad.zahorod...@kde.org>
 > >> wrote:
 >
 >     On 9/5/21 3:18 PM, David Faure wrote:
 >      > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
 >      >> On Sun, Sep 5, 2021 at 10:22 PM David Faure
mailto:fa...@kde.org>
 >     >> wrote:
 >      >>> For frameworks, I think we should be able to write a
one-time
 >     script that
 >      >>> generates .kde-ci.yml files using the dependencies listed in
 >     kde-build-
 >      >>> metadata (and the platforms listed in metainfo.yaml) ?
 >      >>
 >      >> Yes, that should work nicely (although that information
now lives in
 >      >> sysadmin/repo-metadata, dependencies folder)
 >      >
 >      > All done for Frameworks.
 >      >
 >      > The script that collects platforms from metainfo.yaml won't be
 >     useful for
 >      > other KDE modules, but the script that collects deps from
 >     dependency-data-kf5-
 >      > qt5 is attached, in case it's useful to anyone else.
 >
 >     Is there a file that maps legacy project paths to gitlab
project paths?
 >     dependency-data-kf5-qt5 lists projects with their legacy paths.
 >
 >
 > The individual project YAML files in sysadmin/repo-metadata
contain this
 > information.
 > The legacy project path can be found under 'projectpath' while the
 > Gitlab paths are under 'repopath'

Thanks! I've attached a quick and dirty python script that parses
project dependencies from repo-metadata and prints corresponding gitlab
project paths.

Example usage

    python project-deps.py --repo-metadata
/data/projects/src/repo-metadata/ kde/workspace/kwin

Output

Dependencies:
    'requires':
      'frameworks/extra-cmake-modules': '@stable'
      'plasma/kdecoration': '@stable'
      'plasma/kscreenlocker': '@stable'
      'plasma/kwayland-integration': '@stable'
      'plasma/breeze': '@stable'
      'plasma/kwayland-server': '@stable'


Please note that the above is missing a 'on' section as required for 
each Dependencies section in the .kde-ci.yml file.
Yeah... Is there a database or something that can be queried to fill in 
the "on" section?

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 10:14 PM Vlad Zahorodnii 
wrote:

> On 9/7/21 12:22 PM, Ben Cooksley wrote:
> > On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii  > > wrote:
> >
> > On 9/5/21 3:18 PM, David Faure wrote:
> >  > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> >  >> On Sun, Sep 5, 2021 at 10:22 PM David Faure  > > wrote:
> >  >>> For frameworks, I think we should be able to write a one-time
> > script that
> >  >>> generates .kde-ci.yml files using the dependencies listed in
> > kde-build-
> >  >>> metadata (and the platforms listed in metainfo.yaml) ?
> >  >>
> >  >> Yes, that should work nicely (although that information now
> lives in
> >  >> sysadmin/repo-metadata, dependencies folder)
> >  >
> >  > All done for Frameworks.
> >  >
> >  > The script that collects platforms from metainfo.yaml won't be
> > useful for
> >  > other KDE modules, but the script that collects deps from
> > dependency-data-kf5-
> >  > qt5 is attached, in case it's useful to anyone else.
> >
> > Is there a file that maps legacy project paths to gitlab project
> paths?
> > dependency-data-kf5-qt5 lists projects with their legacy paths.
> >
> >
> > The individual project YAML files in sysadmin/repo-metadata contain this
> > information.
> > The legacy project path can be found under 'projectpath' while the
> > Gitlab paths are under 'repopath'
>
> Thanks! I've attached a quick and dirty python script that parses
> project dependencies from repo-metadata and prints corresponding gitlab
> project paths.
>
> Example usage
>
>python project-deps.py --repo-metadata
> /data/projects/src/repo-metadata/ kde/workspace/kwin
>
> Output
>
> Dependencies:
>'requires':
>  'frameworks/extra-cmake-modules': '@stable'
>  'plasma/kdecoration': '@stable'
>  'plasma/kscreenlocker': '@stable'
>  'plasma/kwayland-integration': '@stable'
>  'plasma/breeze': '@stable'
>  'plasma/kwayland-server': '@stable'
>

Please note that the above is missing a 'on' section as required for each
Dependencies section in the .kde-ci.yml file.


> Cheers,
> Vlad
>

Cheers,
Ben


> >
> >
> > Cheers,
> > Ben
> >
> >
> >  >> Once we've transitioned across to the .kde-ci.yml files the
> existing
> >  >> dependency metadata and branch group files will no longer be
> > required.
> >  >> (We'll need to work out a way of exporting this information for
> easy
> >  >> consumption by kdesrc-build and other clients before it can be
> > removed
> >  >> though)
> >  >
> >  > OK. Duplication is bad so I agree :)
> >  >
> >
>
>

Re: Gitlab CI - Inbound

[Vlad Zahorodnii]

On 9/7/21 12:22 PM, Ben Cooksley wrote:
On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii > wrote:


On 9/5/21 3:18 PM, David Faure wrote:
 > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
 >> On Sun, Sep 5, 2021 at 10:22 PM David Faure mailto:fa...@kde.org>> wrote:
 >>> For frameworks, I think we should be able to write a one-time
script that
 >>> generates .kde-ci.yml files using the dependencies listed in
kde-build-
 >>> metadata (and the platforms listed in metainfo.yaml) ?
 >>
 >> Yes, that should work nicely (although that information now lives in
 >> sysadmin/repo-metadata, dependencies folder)
 >
 > All done for Frameworks.
 >
 > The script that collects platforms from metainfo.yaml won't be
useful for
 > other KDE modules, but the script that collects deps from
dependency-data-kf5-
 > qt5 is attached, in case it's useful to anyone else.

Is there a file that maps legacy project paths to gitlab project paths?
dependency-data-kf5-qt5 lists projects with their legacy paths.


The individual project YAML files in sysadmin/repo-metadata contain this 
information.
The legacy project path can be found under 'projectpath' while the 
Gitlab paths are under 'repopath'


Thanks! I've attached a quick and dirty python script that parses 
project dependencies from repo-metadata and prints corresponding gitlab 
project paths.


Example usage

  python project-deps.py --repo-metadata 
/data/projects/src/repo-metadata/ kde/workspace/kwin


Output

Dependencies:
  'requires':
'frameworks/extra-cmake-modules': '@stable'
'plasma/kdecoration': '@stable'
'plasma/kscreenlocker': '@stable'
'plasma/kwayland-integration': '@stable'
'plasma/breeze': '@stable'
'plasma/kwayland-server': '@stable'

Cheers,
Vlad




Cheers,
Ben


 >> Once we've transitioned across to the .kde-ci.yml files the existing
 >> dependency metadata and branch group files will no longer be
required.
 >> (We'll need to work out a way of exporting this information for easy
 >> consumption by kdesrc-build and other clients before it can be
removed
 >> though)
 >
 > OK. Duplication is bad so I agree :)
 >



#!/usr/bin/env python3

import argparse
import glob
import os
import yaml


def build_legacy_to_gitlab_table(repo_metadata):
table = dict()
repo_root = os.path.join(repo_metadata, "projects-invent/**/**/metadata.yaml")
for filename in glob.glob(repo_root):
with open(filename, "r") as f:
document = yaml.safe_load(f)
legacy_path = document["projectpath"]
gitlab_path = document["repopath"]
if not legacy_path:
print(filename + " is invalid")
elif not gitlab_path:
print(filename + " is invalid")
else:
table[legacy_path] = gitlab_path
return table


def parse_dependencies(repo_metadata, legacy_project_path):
deps_file = os.path.join(repo_metadata, "dependencies", "dependency-data-kf5-qt5")
dependencies = list()
needle = legacy_project_path + ":"
with open(deps_file, "r") as f:
for line in f:
if line.startswith("*:"):
dep = line[len("*:"):].strip()
elif line.startswith(needle):
dep = line[len(needle):].strip()
else:
continue
if dep.startswith("-"):
dep = dep[1:]
dependencies.remove(dep)
else:
dependencies.append(dep)
return dependencies


def print_dependencies(mapping_table, dependencies):
gitlab_deps = list()
for dependency in dependencies:
if dependency in mapping_table:
gitlab_deps.append(mapping_table[dependency])
elif dependency != "third-party/Qt5":
print(dependency + " has no matching gitlab repo")
if gitlab_deps:
print("Dependencies:")
print("  'requires':")
for dep in gitlab_deps:
print(f"'{dep}': '@stable'")


parser = argparse.ArgumentParser(description='kde ci dependency builder')
parser.add_argument('--repo-metadata',
help='the file path to sysadmin/repo-metadata repo')
parser.add_argument('projectpath', help='projectpath')
args = parser.parse_args()

repo_metadata = args.repo_metadata
projectpath = args.projectpath

mapping_table = build_legacy_to_gitlab_table(repo_metadata)
legacy_dependencies = parse_dependencies(repo_metadata, projectpath)
print_dependencies(mapping_table, legacy_dependencies)

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 8:48 PM Vlad Zahorodnii 
wrote:

> On 9/5/21 3:18 PM, David Faure wrote:
> > On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> >> On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> >>> For frameworks, I think we should be able to write a one-time script
> that
> >>> generates .kde-ci.yml files using the dependencies listed in kde-build-
> >>> metadata (and the platforms listed in metainfo.yaml) ?
> >>
> >> Yes, that should work nicely (although that information now lives in
> >> sysadmin/repo-metadata, dependencies folder)
> >
> > All done for Frameworks.
> >
> > The script that collects platforms from metainfo.yaml won't be useful for
> > other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> > qt5 is attached, in case it's useful to anyone else.
>
> Is there a file that maps legacy project paths to gitlab project paths?
> dependency-data-kf5-qt5 lists projects with their legacy paths.
>

The individual project YAML files in sysadmin/repo-metadata contain this
information.
The legacy project path can be found under 'projectpath' while the Gitlab
paths are under 'repopath'


> vlad
>

Cheers,
Ben


> >> Once we've transitioned across to the .kde-ci.yml files the existing
> >> dependency metadata and branch group files will no longer be required.
> >> (We'll need to work out a way of exporting this information for easy
> >> consumption by kdesrc-build and other clients before it can be removed
> >> though)
> >
> > OK. Duplication is bad so I agree :)
> >
>
>

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 9:09 PM David Edmundson 
wrote:

> Excellent news!! Thanks very much
>
> > Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories
>
> Does this mean we would like Plasma to wait a while before merging?
> Is it worth us creating the kde-cli files and not merging them so we
> have some test cases ready?
>

You can certainly get the .kde-ci.yml files in position - I do not expect
the format of them to change at this stage.

The big thing that will delay the rollout is determining how best to handle
the situation of when two projects want different versions of the same
dependency.
I'm going to have to figure that out sooner than anticipated in any event
due to Phonon and plasma-wayland-protocols which are both dependencies of
Frameworks (yet which both also require Frameworks).


> David
>

Cheers,
Ben

Re: Gitlab CI - Inbound

[David Edmundson]

Excellent news!! Thanks very much

> Once the scripts have been proven successfully for Frameworks, we will look 
> at extending them to projects that depend only on Frameworks and repositories

Does this mean we would like Plasma to wait a while before merging?
Is it worth us creating the kde-cli files and not merging them so we
have some test cases ready?

David

Re: Gitlab CI - Inbound

[Vlad Zahorodnii]

On 9/5/21 3:18 PM, David Faure wrote:

On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:

On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:

For frameworks, I think we should be able to write a one-time script that
generates .kde-ci.yml files using the dependencies listed in kde-build-
metadata (and the platforms listed in metainfo.yaml) ?


Yes, that should work nicely (although that information now lives in
sysadmin/repo-metadata, dependencies folder)


All done for Frameworks.

The script that collects platforms from metainfo.yaml won't be useful for
other KDE modules, but the script that collects deps from dependency-data-kf5-
qt5 is attached, in case it's useful to anyone else.


Is there a file that maps legacy project paths to gitlab project paths? 
dependency-data-kf5-qt5 lists projects with their legacy paths.


vlad


Once we've transitioned across to the .kde-ci.yml files the existing
dependency metadata and branch group files will no longer be required.
(We'll need to work out a way of exporting this information for easy
consumption by kdesrc-build and other clients before it can be removed
though)


OK. Duplication is bad so I agree :)

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 6:20 AM Johnny Jazeix  wrote:

> Hi Ben,
> not sure on which priority it is regarding the KDE Frameworks but I've
> added one on GCompris (
> https://invent.kde.org/education/gcompris/-/commit/67c9839d7970b360b5d6b0ec928b492f9003d07d)
> if it can help on more tests.
>

Thanks for getting that landed Johnny.

Please note that you've specified no dependencies, so your builds won't
even have ECM available so you may wish to fix that.


> Cheers,
>
> Johnny
>

Cheers,
Ben


> Le dim. 5 sept. 2021 à 12:11, Ben Cooksley  a écrit :
>
>> On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:
>>
>>> Hi all,
>>>
>>
>> Hi all,
>>
>>
>>> This morning after much work i'm happy to announce that the new
>>> generation CI scripts intended for use with Gitlab CI successfully
>>> completed their first build (of ECM, and then subsequently of KCoreAddons).
>>>
>>> This begins our first steps towards transferring CI over from Jenkins to
>>> Gitlab.
>>>
>>> These scripts are 'Gitlab native' and are designed to work in an
>>> environment where they will be running on merge requests, tags as well as
>>> all branches of our repositories - something the existing scripts were
>>> completely incompatible with. While there are still some infrastructural
>>> elements to put in place (such as 'seed jobs' to mass rebuild all projects
>>> after substantial changes and 'cleanup jobs' to remove old builds from the
>>> Package Registry) the core elements needed for initial testing of these
>>> scripts are now ready.
>>>
>>
>> As an update, an initial version of the seed job tooling is now ready,
>> however testing that tooling requires that dependency information be
>> available.
>> This requires that the .kde-ci.yml files be populated in repositories.
>>
>> It would be appreciated if people could please work on getting these
>> files populated in Frameworks (as everyone needs those) as well as in their
>> own repositories as they are required before we can proceed much further.
>>
>>
>>> For those curious, the results of those initials runs can be found at
>>> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>>>
>>> Due to the challenges associated with having to handle all of the
>>> different forms of build and the versioning of this information, these
>>> scripts also represent a radical change in how CI builds will be conducted
>>> - with a large part of the configuration of the jobs themselves, including
>>> information on project dependencies now shifting to files located within
>>> the repository themselves. Those who monitor commits to Frameworks
>>> repositories will have noticed the appearance of these '.kde-ci.yml' files
>>> in some repositories.
>>>
>>> To assist in the future rollout of the CI system it would be appreciated
>>> if projects could please begin setting up these files within their
>>> respective repositories.
>>> You can find a template detailing the available options at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>>>
>>> Where possible please avoid overriding the system defaults except where
>>> needed by your project (ie. don't just copy the template in full)
>>> The defaults mirror the template and can be found at
>>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>>>
>>> In terms of the format of the 'Dependencies' section, please bear in
>>> mind the following:
>>> - For the 'on' section, the special value '@all' can be used to mean
>>> "All platforms" to avoid needing to update the file in the event additional
>>> platforms are added in the future
>>> - For the 'require' section:
>>>   1) Please only include the projects you *explicitly* depend on.
>>> Dependencies of your dependencies will be included automatically
>>>   2) When specifying a project, you must use the repository path on
>>> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
>>> longer in use and will not work.
>>>   3) There are three special values for the branch name - '@same',
>>> '@latest' and '@stable' which can be used to refer to the branch/tag of a
>>> dependency.
>>>   '@same' means the branch name should be the same as that of the
>>> project being built and should be used when declaring dependencies among
>>> projects in a release group.
>>>   '@latest' and '@stable' refer to the corresponding branches
>>> defined in the branch-rules.yml file, which can be found in
>>> sysadmin/repo-metadata
>>>
>>> As a general rule, unless you require the absolute latest version of
>>> another project in another release unit, it is recommended that you use
>>> @stable.
>>> Please avoid using explicit branch names unless absolutely necessary.
>>>
>>> At this time it is expected that the first set of Gitlab CI builds using
>>> the new scripts may be conducted for Frameworks within the next week or so,
>>> assuming the build of the seed jobs goes according to plan.
>>>
>>> Once the 

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Tue, Sep 7, 2021 at 1:04 AM Tom Zander  wrote:

> On maandag 6 september 2021 11:48:39 CEST Ben Cooksley wrote:
> > > Pushing everything into required is likely not scalable,
> > > causing projects too wait too long for compile.
> > > Avoiding the optional ones means you lack coverage of compile
> > > and testing failures due to changes in libs.
> >
> > The CI system has reused the results of previous builds of
> > dependencies since the very first generation of the system
>
> We seem to be talking about two slightly different topics.
>
> When the (for instance) KIO repo changes, then the CI will
> obviously rebuild that repo and will pull in all the things that
> kio depends on.
>
> What many CIs do is additionally trigger rebuilds of projects
> that _use_ KIO, by them marking kio as a required dependency.
>

Our CI system has never done this, in part due to difficulties in
communicating this to Jenkins and also because of the build storm it would
create (which I believe is what you are referring to).
If we were to limit it to select projects, then we would need to 'pick'
those projects which would raise questions of favouritism which I'd rather
avoid.


> Imagine a small extragear app that uses some KIO stuff and it has
> some unit tests that would break as a result of the KIO change.
> When the KDE CI triggers a rebuild of projects that mark KIO as a
> required dependency, this little app would show its breaking as a
> result of the KIO push. Helping the KIO dev to realize the
> fallout as well.
> Without such a feature the app would show breakage at a random
> time in the future after a new push was made in that repo. Losing
> lots of dev time and compromising quality.
>
> Now, the optional requirements would help diminish the effect of
> changes in frameworks paralysing the build system by limiting the
> apps that gets scheduled for a rebuild.
>

> This kind of functionality becomes pretty easy to add to gen5.1
> of the CI, provided that at this time the dependencies are
> already split since doing it later is going to be an uphill
> battle.
>

Cheers,
Ben

Re: Gitlab CI - Inbound

[Johnny Jazeix]

Hi Ben,
not sure on which priority it is regarding the KDE Frameworks but I've
added one on GCompris (
https://invent.kde.org/education/gcompris/-/commit/67c9839d7970b360b5d6b0ec928b492f9003d07d)
if it can help on more tests.

Cheers,

Johnny

Le dim. 5 sept. 2021 à 12:11, Ben Cooksley  a écrit :

> On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:
>
>> Hi all,
>>
>
> Hi all,
>
>
>> This morning after much work i'm happy to announce that the new
>> generation CI scripts intended for use with Gitlab CI successfully
>> completed their first build (of ECM, and then subsequently of KCoreAddons).
>>
>> This begins our first steps towards transferring CI over from Jenkins to
>> Gitlab.
>>
>> These scripts are 'Gitlab native' and are designed to work in an
>> environment where they will be running on merge requests, tags as well as
>> all branches of our repositories - something the existing scripts were
>> completely incompatible with. While there are still some infrastructural
>> elements to put in place (such as 'seed jobs' to mass rebuild all projects
>> after substantial changes and 'cleanup jobs' to remove old builds from the
>> Package Registry) the core elements needed for initial testing of these
>> scripts are now ready.
>>
>
> As an update, an initial version of the seed job tooling is now ready,
> however testing that tooling requires that dependency information be
> available.
> This requires that the .kde-ci.yml files be populated in repositories.
>
> It would be appreciated if people could please work on getting these files
> populated in Frameworks (as everyone needs those) as well as in their own
> repositories as they are required before we can proceed much further.
>
>
>> For those curious, the results of those initials runs can be found at
>> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>>
>> Due to the challenges associated with having to handle all of the
>> different forms of build and the versioning of this information, these
>> scripts also represent a radical change in how CI builds will be conducted
>> - with a large part of the configuration of the jobs themselves, including
>> information on project dependencies now shifting to files located within
>> the repository themselves. Those who monitor commits to Frameworks
>> repositories will have noticed the appearance of these '.kde-ci.yml' files
>> in some repositories.
>>
>> To assist in the future rollout of the CI system it would be appreciated
>> if projects could please begin setting up these files within their
>> respective repositories.
>> You can find a template detailing the available options at
>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>>
>> Where possible please avoid overriding the system defaults except where
>> needed by your project (ie. don't just copy the template in full)
>> The defaults mirror the template and can be found at
>> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>>
>> In terms of the format of the 'Dependencies' section, please bear in mind
>> the following:
>> - For the 'on' section, the special value '@all' can be used to mean "All
>> platforms" to avoid needing to update the file in the event additional
>> platforms are added in the future
>> - For the 'require' section:
>>   1) Please only include the projects you *explicitly* depend on.
>> Dependencies of your dependencies will be included automatically
>>   2) When specifying a project, you must use the repository path on
>> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
>> longer in use and will not work.
>>   3) There are three special values for the branch name - '@same',
>> '@latest' and '@stable' which can be used to refer to the branch/tag of a
>> dependency.
>>   '@same' means the branch name should be the same as that of the
>> project being built and should be used when declaring dependencies among
>> projects in a release group.
>>   '@latest' and '@stable' refer to the corresponding branches defined
>> in the branch-rules.yml file, which can be found in sysadmin/repo-metadata
>>
>> As a general rule, unless you require the absolute latest version of
>> another project in another release unit, it is recommended that you use
>> @stable.
>> Please avoid using explicit branch names unless absolutely necessary.
>>
>> At this time it is expected that the first set of Gitlab CI builds using
>> the new scripts may be conducted for Frameworks within the next week or so,
>> assuming the build of the seed jobs goes according to plan.
>>
>> Once the scripts have been proven successfully for Frameworks, we will
>> look at extending them to projects that depend only on Frameworks and
>> repositories within their release unit and finally will extend them to
>> projects with more complex dependency requirements. It is expected that the
>> switch will be flipped on the Frameworks builds sometime in the 

Re: Gitlab CI - Inbound

[Tom Zander]

On maandag 6 september 2021 14:21:00 CEST Christoph Cullmann 
(cullmann.io) wrote:
> I would prefer to not have some optional category to be sure
> the CI always builds with the same state of dependencies, too.

A CI build would always build with the latest release (or 
whatever the repo owner stated). The "optional" would have zero 
effect on that.

Re: Gitlab CI - Inbound

[Tom Zander]

On maandag 6 september 2021 11:48:39 CEST Ben Cooksley wrote:
> > Pushing everything into required is likely not scalable,
> > causing projects too wait too long for compile.
> > Avoiding the optional ones means you lack coverage of compile
> > and testing failures due to changes in libs.
> 
> The CI system has reused the results of previous builds of
> dependencies since the very first generation of the system

We seem to be talking about two slightly different topics.

When the (for instance) KIO repo changes, then the CI will 
obviously rebuild that repo and will pull in all the things that 
kio depends on.

What many CIs do is additionally trigger rebuilds of projects 
that _use_ KIO, by them marking kio as a required dependency.

Imagine a small extragear app that uses some KIO stuff and it has 
some unit tests that would break as a result of the KIO change.
When the KDE CI triggers a rebuild of projects that mark KIO as a 
required dependency, this little app would show its breaking as a 
result of the KIO push. Helping the KIO dev to realize the 
fallout as well.
Without such a feature the app would show breakage at a random 
time in the future after a new push was made in that repo. Losing 
lots of dev time and compromising quality.

Now, the optional requirements would help diminish the effect of 
changes in frameworks paralysing the build system by limiting the 
apps that gets scheduled for a rebuild.

This kind of functionality becomes pretty easy to add to gen5.1 
of the CI, provided that at this time the dependencies are 
already split since doing it later is going to be an uphill 
battle.

Re: Gitlab CI - Inbound

[Tom Zander]

On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> In terms of the format of the 'Dependencies' section,

Playing with kde-build script and noticing the fast growing 
dependency trees we have today, I think it may be beneficial to 
have two types of compile dependencies in this setup.

1. required-to-build.
  Which means that if something in the parent tree changes, this
  one is scheduled for re-build.

2. optional. Equivalent of the cmake feature, if its not there
  some code is not compiled.
  At least once before a release the full dependencies can be
  compiled to see if it fully compiles.


Pushing everything into required is likely not scalable, causing 
projects too wait too long for compile.
Avoiding the optional ones means you lack coverage of compile and 
testing failures due to changes in libs.

What do people think, is it useful to have an 'optional' category 
in future there?   
Maybe useful to think that far ahead now people are populating 
their dependencies :-)

Re: Gitlab CI - Inbound

[Michael Reeves]

How do we get a visual on exactly which lines are covered by auto testing
and which aren't?

Re: Gitlab CI - Inbound

[Christoph Cullmann (cullmann.io)]

On 2021-09-06 11:48, Ben Cooksley wrote:

On Mon, Sep 6, 2021 at 9:00 PM Tom Zander  wrote:


On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:

In terms of the format of the 'Dependencies' section,


Playing with kde-build script and noticing the fast growing
dependency trees we have today, I think it may be beneficial to
have two types of compile dependencies in this setup.

1. required-to-build.
Which means that if something in the parent tree changes, this
one is scheduled for re-build.

2. optional. Equivalent of the cmake feature, if its not there
some code is not compiled.
At least once before a release the full dependencies can be
compiled to see if it fully compiles.


From the perspective of the CI system there is basically no difference
in terms of making a dependency available or not as all dependencies
are always satisfied using previously built binaries.
If a dependency is not available your build fails.

We also have to make a hard choice - we either bring in optional
dependencies or we don't.

If we were to randomise whether we brought them in - or just brought
them in at certain times - then we would make the system state
deterministic. This could in some cases cause builds to break if it
was random whether or not we included optional dependencies. This
would occur if the dependency of a dependency of a project were
rebuilt without an optional dependency being present which in turn
caused it's API interface to change. That would then cause the project
to fail as the dependency would now be broken.


Pushing everything into required is likely not scalable, causing
projects too wait too long for compile.
Avoiding the optional ones means you lack coverage of compile and
testing failures due to changes in libs.


The CI system has reused the results of previous builds of
dependencies since the very first generation of the system (this would
be generation 5) so this is not a problem facing the CI system.
For individual developers, my understanding is that kdesrc-build makes
use of incremental builds which eliminates most of the issue as well.


What do people think, is it useful to have an 'optional' category
in future there?
Maybe useful to think that far ahead now people are populating
their dependencies :-)


Hi,

I would prefer to not have some optional category to be sure
the CI always builds with the same state of dependencies, too.

Greetings
Christoph

--
Ignorance is bliss...
https://cullmann.io | https://kate-editor.org

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 9:00 PM Tom Zander  wrote:

> On zondag 5 september 2021 08:13:09 CEST Ben Cooksley wrote:
> > In terms of the format of the 'Dependencies' section,
>
> Playing with kde-build script and noticing the fast growing
> dependency trees we have today, I think it may be beneficial to
> have two types of compile dependencies in this setup.
>
> 1. required-to-build.
>   Which means that if something in the parent tree changes, this
>   one is scheduled for re-build.
>
> 2. optional. Equivalent of the cmake feature, if its not there
>   some code is not compiled.
>   At least once before a release the full dependencies can be
>   compiled to see if it fully compiles.
>

>From the perspective of the CI system there is basically no difference in
terms of making a dependency available or not as all dependencies are
always satisfied using previously built binaries.
If a dependency is not available your build fails.

We also have to make a hard choice - we either bring in optional
dependencies or we don't.

If we were to randomise whether we brought them in - or just brought them
in at certain times - then we would make the system state deterministic.
This could in some cases cause builds to break if it was random whether or
not we included optional dependencies. This would occur if the dependency
of a dependency of a project were rebuilt without an optional dependency
being present which in turn caused it's API interface to change. That would
then cause the project to fail as the dependency would now be broken.


>
> Pushing everything into required is likely not scalable, causing
> projects too wait too long for compile.
> Avoiding the optional ones means you lack coverage of compile and
> testing failures due to changes in libs.
>

The CI system has reused the results of previous builds of dependencies
since the very first generation of the system (this would be generation 5)
so this is not a problem facing the CI system.
For individual developers, my understanding is that kdesrc-build makes use
of incremental builds which eliminates most of the issue as well.


> What do people think, is it useful to have an 'optional' category
> in future there?
> Maybe useful to think that far ahead now people are populating
> their dependencies :-)
>
>
>
>
Thanks,
Ben

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:18 AM David Faure  wrote:

> On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> > On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > > For frameworks, I think we should be able to write a one-time script
> that
> > > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > > metadata (and the platforms listed in metainfo.yaml) ?
> >
> > Yes, that should work nicely (although that information now lives in
> > sysadmin/repo-metadata, dependencies folder)
>
> All done for Frameworks.
>
> The script that collects platforms from metainfo.yaml won't be useful for
> other KDE modules, but the script that collects deps from
> dependency-data-kf5-
> qt5 is attached, in case it's useful to anyone else.
>

Thanks for sorting this David, very much appreciated.


> > Once we've transitioned across to the .kde-ci.yml files the existing
> > dependency metadata and branch group files will no longer be required.
> > (We'll need to work out a way of exporting this information for easy
> > consumption by kdesrc-build and other clients before it can be removed
> > though)
>
> OK. Duplication is bad so I agree :)
>

Cheers,
Ben


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 12:46 AM Nicolas Fella  wrote:

> On 05.09.21 08:13, Ben Cooksley wrote:
> > Hi all,
> >
> > This morning after much work i'm happy to announce that the new
> > generation CI scripts intended for use with Gitlab CI successfully
> > completed their first build (of ECM, and then subsequently of
> > KCoreAddons).
> >
> > This begins our first steps towards transferring CI over from Jenkins
> > to Gitlab.
> >
> > These scripts are 'Gitlab native' and are designed to work in an
> > environment where they will be running on merge requests, tags as well
> > as all branches of our repositories - something the existing scripts
> > were completely incompatible with. While there are still some
> > infrastructural elements to put in place (such as 'seed jobs' to mass
> > rebuild all projects after substantial changes and 'cleanup jobs' to
> > remove old builds from the Package Registry) the core elements needed
> > for initial testing of these scripts are now ready.
> >
> > For those curious, the results of those initials runs can be found at
> > https://invent.kde.org/groups/teams/ci-artifacts/-/packages
> > 
> >
> > Due to the challenges associated with having to handle all of the
> > different forms of build and the versioning of this information, these
> > scripts also represent a radical change in how CI builds will be
> > conducted - with a large part of the configuration of the jobs
> > themselves, including information on project dependencies now shifting
> > to files located within the repository themselves. Those who monitor
> > commits to Frameworks repositories will have noticed the appearance of
> > these '.kde-ci.yml' files in some repositories.
> >
> > To assist in the future rollout of the CI system it would be
> > appreciated if projects could please begin setting up these files
> > within their respective repositories.
> > You can find a template detailing the available options at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
> >
> >
> > Where possible please avoid overriding the system defaults except
> > where needed by your project (ie. don't just copy the template in full)
> > The defaults mirror the template and can be found at
> >
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> > <
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
> >
> >
> > In terms of the format of the 'Dependencies' section, please bear in
> > mind the following:
> > - For the 'on' section, the special value '@all' can be used to mean
> > "All platforms" to avoid needing to update the file in the event
> > additional platforms are added in the future
> > - For the 'require' section:
> >   1) Please only include the projects you *explicitly* depend on.
> > Dependencies of your dependencies will be included automatically
> >   2) When specifying a project, you must use the repository path on
> > Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
> > are no longer in use and will not work.
> >   3) There are three special values for the branch name - '@same',
> > '@latest' and '@stable' which can be used to refer to the branch/tag
> > of a dependency.
> >   '@same' means the branch name should be the same as that of the
> > project being built and should be used when declaring dependencies
> > among projects in a release group.
> >   '@latest' and '@stable' refer to the corresponding branches
> > defined in the branch-rules.yml file, which can be found in
> > sysadmin/repo-metadata
> >
> > As a general rule, unless you require the absolute latest version of
> > another project in another release unit, it is recommended that you
> > use @stable.
> > Please avoid using explicit branch names unless absolutely necessary.
> >
> > At this time it is expected that the first set of Gitlab CI builds
> > using the new scripts may be conducted for Frameworks within the next
> > week or so, assuming the build of the seed jobs goes according to plan.
> >
> > Once the scripts have been proven successfully for Frameworks, we will
> > look at extending them to projects that depend only on Frameworks and
> > repositories within their release unit and finally will extend them to
> > projects with more complex dependency requirements. It is expected
> > that the switch will be flipped on the Frameworks builds sometime in
> > the next week or two.
> >
> > Please let me know if you have any questions on the above.
> >
> > Thanks,
> > Ben Cooksley
> > KDE Sysadmin
>
> Hi Ben,
>

Hi Nicolas,


> thanks for your work on this!
>
> How would I best encode a dependency like "On Linux and FreeBSD depend
> on kwayland, but not on Windows and macOS"?
>

I would suggest doing something like the 

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Mon, Sep 6, 2021 at 11:03 AM Michael Reeves  wrote:

> How do we get a visual on exactly which lines are covered by auto testing
> and which aren't?
>

Please see
https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
for more details on how this works on Merge Requests.
To my knowledge this isn't exposed outside of Merge Requests.

Cheers,
Ben

Re: Gitlab CI - Inbound

[Nicolas Fella]

On 05.09.21 08:13, Ben Cooksley wrote:

Hi all,

This morning after much work i'm happy to announce that the new
generation CI scripts intended for use with Gitlab CI successfully
completed their first build (of ECM, and then subsequently of
KCoreAddons).

This begins our first steps towards transferring CI over from Jenkins
to Gitlab.

These scripts are 'Gitlab native' and are designed to work in an
environment where they will be running on merge requests, tags as well
as all branches of our repositories - something the existing scripts
were completely incompatible with. While there are still some
infrastructural elements to put in place (such as 'seed jobs' to mass
rebuild all projects after substantial changes and 'cleanup jobs' to
remove old builds from the Package Registry) the core elements needed
for initial testing of these scripts are now ready.

For those curious, the results of those initials runs can be found at
https://invent.kde.org/groups/teams/ci-artifacts/-/packages


Due to the challenges associated with having to handle all of the
different forms of build and the versioning of this information, these
scripts also represent a radical change in how CI builds will be
conducted - with a large part of the configuration of the jobs
themselves, including information on project dependencies now shifting
to files located within the repository themselves. Those who monitor
commits to Frameworks repositories will have noticed the appearance of
these '.kde-ci.yml' files in some repositories.

To assist in the future rollout of the CI system it would be
appreciated if projects could please begin setting up these files
within their respective repositories.
You can find a template detailing the available options at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml


Where possible please avoid overriding the system defaults except
where needed by your project (ie. don't just copy the template in full)
The defaults mirror the template and can be found at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml


In terms of the format of the 'Dependencies' section, please bear in
mind the following:
- For the 'on' section, the special value '@all' can be used to mean
"All platforms" to avoid needing to update the file in the event
additional platforms are added in the future
- For the 'require' section:
  1) Please only include the projects you *explicitly* depend on.
Dependencies of your dependencies will be included automatically
  2) When specifying a project, you must use the repository path on
Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop)
are no longer in use and will not work.
  3) There are three special values for the branch name - '@same',
'@latest' and '@stable' which can be used to refer to the branch/tag
of a dependency.
  '@same' means the branch name should be the same as that of the
project being built and should be used when declaring dependencies
among projects in a release group.
  '@latest' and '@stable' refer to the corresponding branches
defined in the branch-rules.yml file, which can be found in
sysadmin/repo-metadata

As a general rule, unless you require the absolute latest version of
another project in another release unit, it is recommended that you
use @stable.
Please avoid using explicit branch names unless absolutely necessary.

At this time it is expected that the first set of Gitlab CI builds
using the new scripts may be conducted for Frameworks within the next
week or so, assuming the build of the seed jobs goes according to plan.

Once the scripts have been proven successfully for Frameworks, we will
look at extending them to projects that depend only on Frameworks and
repositories within their release unit and finally will extend them to
projects with more complex dependency requirements. It is expected
that the switch will be flipped on the Frameworks builds sometime in
the next week or two.

Please let me know if you have any questions on the above.

Thanks,
Ben Cooksley
KDE Sysadmin


Hi Ben,

thanks for your work on this!

How would I best encode a dependency like "On Linux and FreeBSD depend
on kwayland, but not on Windows and macOS"?

Cheers

Nico

Re: Gitlab CI - Inbound

[David Faure]

On dimanche 5 septembre 2021 12:26:50 CEST Ben Cooksley wrote:
> On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:
> > For frameworks, I think we should be able to write a one-time script that
> > generates .kde-ci.yml files using the dependencies listed in kde-build-
> > metadata (and the platforms listed in metainfo.yaml) ?
> 
> Yes, that should work nicely (although that information now lives in
> sysadmin/repo-metadata, dependencies folder)

All done for Frameworks.

The script that collects platforms from metainfo.yaml won't be useful for 
other KDE modules, but the script that collects deps from dependency-data-kf5-
qt5 is attached, in case it's useful to anyone else.

> Once we've transitioned across to the .kde-ci.yml files the existing
> dependency metadata and branch group files will no longer be required.
> (We'll need to work out a way of exporting this information for easy
> consumption by kdesrc-build and other clients before it can be removed
> though)

OK. Duplication is bad so I agree :)

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5


collect_deps.pl
Description: Perl program

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 10:22 PM David Faure  wrote:

> On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> > It would be appreciated if people could please work on getting these
> files
> > populated in Frameworks (as everyone needs those) as well as in their own
> > repositories as they are required before we can proceed much further.
>
> Like this
> https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?
>

Yes - just two small tweaks needed for that one but it is otherwise good to
ship.


> For frameworks, I think we should be able to write a one-time script that
> generates .kde-ci.yml files using the dependencies listed in kde-build-
> metadata (and the platforms listed in metainfo.yaml) ?
>

Yes, that should work nicely (although that information now lives in
sysadmin/repo-metadata, dependencies folder)

Once we've transitioned across to the .kde-ci.yml files the existing
dependency metadata and branch group files will no longer be required.
(We'll need to work out a way of exporting this information for easy
consumption by kdesrc-build and other clients before it can be removed
though)


> --
> David Faure, fa...@kde.org, http://www.davidfaure.fr
> Working on KDE Frameworks 5
>
>
Thanks,
Ben

Re: Gitlab CI - Inbound

[David Faure]

On dimanche 5 septembre 2021 12:11:05 CEST Ben Cooksley wrote:
> It would be appreciated if people could please work on getting these files
> populated in Frameworks (as everyone needs those) as well as in their own
> repositories as they are required before we can proceed much further.

Like this https://invent.kde.org/frameworks/kcoreaddons/-/merge_requests/135 ?

For frameworks, I think we should be able to write a one-time script that 
generates .kde-ci.yml files using the dependencies listed in kde-build-
metadata (and the platforms listed in metainfo.yaml) ?

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

Re: Gitlab CI - Inbound

[Ben Cooksley]

On Sun, Sep 5, 2021 at 6:13 PM Ben Cooksley  wrote:

> Hi all,
>

Hi all,


> This morning after much work i'm happy to announce that the new generation
> CI scripts intended for use with Gitlab CI successfully completed their
> first build (of ECM, and then subsequently of KCoreAddons).
>
> This begins our first steps towards transferring CI over from Jenkins to
> Gitlab.
>
> These scripts are 'Gitlab native' and are designed to work in an
> environment where they will be running on merge requests, tags as well as
> all branches of our repositories - something the existing scripts were
> completely incompatible with. While there are still some infrastructural
> elements to put in place (such as 'seed jobs' to mass rebuild all projects
> after substantial changes and 'cleanup jobs' to remove old builds from the
> Package Registry) the core elements needed for initial testing of these
> scripts are now ready.
>

As an update, an initial version of the seed job tooling is now ready,
however testing that tooling requires that dependency information be
available.
This requires that the .kde-ci.yml files be populated in repositories.

It would be appreciated if people could please work on getting these files
populated in Frameworks (as everyone needs those) as well as in their own
repositories as they are required before we can proceed much further.


> For those curious, the results of those initials runs can be found at
> https://invent.kde.org/groups/teams/ci-artifacts/-/packages
>
> Due to the challenges associated with having to handle all of the
> different forms of build and the versioning of this information, these
> scripts also represent a radical change in how CI builds will be conducted
> - with a large part of the configuration of the jobs themselves, including
> information on project dependencies now shifting to files located within
> the repository themselves. Those who monitor commits to Frameworks
> repositories will have noticed the appearance of these '.kde-ci.yml' files
> in some repositories.
>
> To assist in the future rollout of the CI system it would be appreciated
> if projects could please begin setting up these files within their
> respective repositories.
> You can find a template detailing the available options at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml
>
> Where possible please avoid overriding the system defaults except where
> needed by your project (ie. don't just copy the template in full)
> The defaults mirror the template and can be found at
> https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml
>
> In terms of the format of the 'Dependencies' section, please bear in mind
> the following:
> - For the 'on' section, the special value '@all' can be used to mean "All
> platforms" to avoid needing to update the file in the event additional
> platforms are added in the future
> - For the 'require' section:
>   1) Please only include the projects you *explicitly* depend on.
> Dependencies of your dependencies will be included automatically
>   2) When specifying a project, you must use the repository path on
> Gitlab. Legacy project paths (such as kde/workspace/plasma-desktop) are no
> longer in use and will not work.
>   3) There are three special values for the branch name - '@same',
> '@latest' and '@stable' which can be used to refer to the branch/tag of a
> dependency.
>   '@same' means the branch name should be the same as that of the
> project being built and should be used when declaring dependencies among
> projects in a release group.
>   '@latest' and '@stable' refer to the corresponding branches defined
> in the branch-rules.yml file, which can be found in sysadmin/repo-metadata
>
> As a general rule, unless you require the absolute latest version of
> another project in another release unit, it is recommended that you use
> @stable.
> Please avoid using explicit branch names unless absolutely necessary.
>
> At this time it is expected that the first set of Gitlab CI builds using
> the new scripts may be conducted for Frameworks within the next week or so,
> assuming the build of the seed jobs goes according to plan.
>
> Once the scripts have been proven successfully for Frameworks, we will
> look at extending them to projects that depend only on Frameworks and
> repositories within their release unit and finally will extend them to
> projects with more complex dependency requirements. It is expected that the
> switch will be flipped on the Frameworks builds sometime in the next week
> or two.
>
> Please let me know if you have any questions on the above.
>
> Thanks,
> Ben Cooksley
> KDE Sysadmin
>

Thanks,
Ben

Gitlab CI - Inbound

[Ben Cooksley]

Hi all,

This morning after much work i'm happy to announce that the new generation
CI scripts intended for use with Gitlab CI successfully completed their
first build (of ECM, and then subsequently of KCoreAddons).

This begins our first steps towards transferring CI over from Jenkins to
Gitlab.

These scripts are 'Gitlab native' and are designed to work in an
environment where they will be running on merge requests, tags as well as
all branches of our repositories - something the existing scripts were
completely incompatible with. While there are still some infrastructural
elements to put in place (such as 'seed jobs' to mass rebuild all projects
after substantial changes and 'cleanup jobs' to remove old builds from the
Package Registry) the core elements needed for initial testing of these
scripts are now ready.

For those curious, the results of those initials runs can be found at
https://invent.kde.org/groups/teams/ci-artifacts/-/packages

Due to the challenges associated with having to handle all of the different
forms of build and the versioning of this information, these scripts also
represent a radical change in how CI builds will be conducted - with a
large part of the configuration of the jobs themselves, including
information on project dependencies now shifting to files located within
the repository themselves. Those who monitor commits to Frameworks
repositories will have noticed the appearance of these '.kde-ci.yml' files
in some repositories.

To assist in the future rollout of the CI system it would be appreciated if
projects could please begin setting up these files within their respective
repositories.
You can find a template detailing the available options at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config-template.yml

Where possible please avoid overriding the system defaults except where
needed by your project (ie. don't just copy the template in full)
The defaults mirror the template and can be found at
https://invent.kde.org/sysadmin/ci-tooling/-/blob/work/bcooksley/gitlab-ci/config/global.yml

In terms of the format of the 'Dependencies' section, please bear in mind
the following:
- For the 'on' section, the special value '@all' can be used to mean "All
platforms" to avoid needing to update the file in the event additional
platforms are added in the future
- For the 'require' section:
  1) Please only include the projects you *explicitly* depend on.
Dependencies of your dependencies will be included automatically
  2) When specifying a project, you must use the repository path on Gitlab.
Legacy project paths (such as kde/workspace/plasma-desktop) are no longer
in use and will not work.
  3) There are three special values for the branch name - '@same',
'@latest' and '@stable' which can be used to refer to the branch/tag of a
dependency.
  '@same' means the branch name should be the same as that of the
project being built and should be used when declaring dependencies among
projects in a release group.
  '@latest' and '@stable' refer to the corresponding branches defined
in the branch-rules.yml file, which can be found in sysadmin/repo-metadata

As a general rule, unless you require the absolute latest version of
another project in another release unit, it is recommended that you use
@stable.
Please avoid using explicit branch names unless absolutely necessary.

At this time it is expected that the first set of Gitlab CI builds using
the new scripts may be conducted for Frameworks within the next week or so,
assuming the build of the seed jobs goes according to plan.

Once the scripts have been proven successfully for Frameworks, we will look
at extending them to projects that depend only on Frameworks and
repositories within their release unit and finally will extend them to
projects with more complex dependency requirements. It is expected that the
switch will be flipped on the Frameworks builds sometime in the next week
or two.

Please let me know if you have any questions on the above.

Thanks,
Ben Cooksley
KDE Sysadmin