Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Stephen Gallagher]

On Fri, Oct 6, 2023 at 11:16 AM Stephen Gallagher  wrote:
...
> So, as we all know, build ordering is hard (and, despite intuitive
> belief, not actually deterministic).
>
> ELN actually "cheats" somewhat when we do our builds. When we process
> a batch of builds (triggered by a set of tag events that come in all
> at the same time, such as when a side-tag is merged), we create a new
> ELN side-tag, tag all of the new *Rawhide* builds into this side tag,
> then trigger a rebuild of all of those builds for ELN. The result here
> is that we use the Fedora build in the buildroot to avoid
> bootstrapping issues. Now, there are some special-case packages for
> which we do *NOT* automatically tag in the Fedora builds because they
> have known incompatibilities. All OCAML packages fall into this
> category, since we discovered about 9 months ago that we absolutely
> cannot mix Fedora's OCAML builds with ELN's (I don't recall the exact
> reason).
>
> Our other "cheat" when we rebuild a batch is that we automatically do
> rebuilds at least once for any failure, to account for things like
> build ordering and test flakes. In the case you're describing,
> unforunately, we had a situation where 1) it was OCAML, and therefore
> the Fedora packages weren't in the buildroot and 2) ocaml built
> successfully against the older version of the macros. If the
> BuildRequires: had been in play there, it would have failed, the batch
> would have finished building whatever else was able to succeed (such
> as the macros) and then the second pass would have succeeded.
>
> I'm sorry you got hit by this, Richard. It's an unfortunate confluence
> of limitations in our rebuild approach.


I thought I'd responded here the other day with the link, but I forgot:
https://sgallagh.wordpress.com/2023/10/13/sausage-factory-fedora-eln-rebuild-strategy/

I've put together a blog post describing our ELN rebuild strategy,
which I'll copy below (but clarifications or additional content may be
added later, so consider that link the most up-to-date version).


---

# Fedora ELN Rebuild Strategy: The Rebuild Algorithm (2023 Edition)

## Slow and Steady Wins the Race

The Fedora ELN SIG maintains a tool called ELNBuildSync[1] (or EBS)
which is responsible for monitoring traffic on the Fedora Messaging
Bus and listening for Koji tagging events. When a package is tagged
into Rawhide (meaning it has passed Fedora QA Gating and is headed to
the official repositories), EBS checks whether it’s on the list of
packages targeted for Fedora ELN or ELN Extras and enqueues it for the
next batch of builds.

A batch begins when there are one or more enqueued builds and at least
five wallclock seconds have passed since a build has been enqueued.
This allows EBS to capture events such as a complete side-tag being
merged into Rawhide at once; it will always rebuild those together in
a batch. Once a batch begins, all other messages are enqueued for the
following batch. When the current batch is complete, a new batch will
begin.

The first thing that is done when processing a batch is to create a
new side-tag derived from the ELN buildroot. Into this new target, EBS
will tag most of the Rawhide builds. It will then wait until Koji has
regenerated the buildroot for the batch tag before triggering the
rebuild of the batched packages. This strategy avoids most of the
ordering issues (particularly bootstrap loops) inherent in rebuilding
a side-tag, because we can rely on the Rawhide builds having already
succeeded.

Once the rebuild is ready to begin, EBS interrogates Koji for the
original git commit used to build each Rawhide package (in case git
has seen subsequent, unbuilt changes). The builds are then triggered
in the side tag concurrently. EBS monitors these builds for
completion. If one or more builds in a batch fails, EBS will re-queue
it for another rebuild attempt. This repeats until the same set of
failures occurs twice in a row. Once all of the rebuild attempts have
concluded, EBS tags all successful builds back to ELN and removes the
side tag. Then it moves on to preparing another batch, if there are
packages waiting.

## History

In its first incarnation, ELNBuildSync (at the time known as
DistroBuildSync) was very simplistic. It listened for tag events on
Rawhide, checked them against its list and then triggered a build in
the ELN target. Very quickly, the ELN SIG realized that this had
significant limitations, particularly in the case of packages building
in side-tags (which was becoming more common as the era of on-demand
side-tags began). One of the main benefits of side-tags is the ability
to rebuild packages that depend on one another in the proper order;
this was lost in the BuildSync process and many times builds were
happening out of order, resulting in packages with the same NVR as
Rawhide but incorrectly built against older versions of their
dependencies.

Initially, the ELN SIG tried to design a way to exactly mirror the
build process in the 

Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Stephen Smoogen]

On Fri, 6 Oct 2023 at 15:45, Richard W.M. Jones  wrote:

> On Fri, Oct 06, 2023 at 11:16:17AM -0400, Stephen Gallagher wrote:
> > So, as we all know, build ordering is hard (and, despite intuitive
> > belief, not actually deterministic).
> >
> > ELN actually "cheats" somewhat when we do our builds. When we process
> > a batch of builds (triggered by a set of tag events that come in all
> > at the same time, such as when a side-tag is merged), we create a new
> > ELN side-tag, tag all of the new *Rawhide* builds into this side tag,
> > then trigger a rebuild of all of those builds for ELN. The result here
> > is that we use the Fedora build in the buildroot to avoid
> > bootstrapping issues. Now, there are some special-case packages for
> > which we do *NOT* automatically tag in the Fedora builds because they
> > have known incompatibilities. All OCAML packages fall into this
> > category, since we discovered about 9 months ago that we absolutely
> > cannot mix Fedora's OCAML builds with ELN's (I don't recall the exact
> > reason).
>
> It's probably because the build flags are different and
> (unnecessarily) OCaml encodes the build flags into the hashes of one
> of the base modules.  I forget the exact details but there's an email
> on this list about it somewhere.  Anyway the upshot is that if the
> build flags change at all, as they obviously do in Fedora vs RHEL,
> then the hashes change.  We ought to fix this upstream ...
>
> Anyway I was going to say why not order the packages by their original
> build date in Koji?
>

Mass rebuilds and multiple developers can make this much harder than
expected. The history looks linear, but it may be that the bootstrap blew
up and had to be restarted OR it does need to do a double loop. Or
developer A rebuilt something when it wasn't needed while B was doing
something in a side tag at the same time. Or you find that rust or go
needed to have old/new mixes which weren't needed last time because
ordering didn't matter that time but does now (this can be due to flags or
package differences.) Many of these things are things which the regular
package maintainers know to do with their pets, but other people do not.

-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Stephen Gallagher]

On Fri, Oct 6, 2023 at 3:45 PM Richard W.M. Jones  wrote:

> On Fri, Oct 06, 2023 at 11:16:17AM -0400, Stephen Gallagher wrote:
> > So, as we all know, build ordering is hard (and, despite intuitive
> > belief, not actually deterministic).
> >
> > ELN actually "cheats" somewhat when we do our builds. When we process
> > a batch of builds (triggered by a set of tag events that come in all
> > at the same time, such as when a side-tag is merged), we create a new
> > ELN side-tag, tag all of the new *Rawhide* builds into this side tag,
> > then trigger a rebuild of all of those builds for ELN. The result here
> > is that we use the Fedora build in the buildroot to avoid
> > bootstrapping issues. Now, there are some special-case packages for
> > which we do *NOT* automatically tag in the Fedora builds because they
> > have known incompatibilities. All OCAML packages fall into this
> > category, since we discovered about 9 months ago that we absolutely
> > cannot mix Fedora's OCAML builds with ELN's (I don't recall the exact
> > reason).
>
> It's probably because the build flags are different and
> (unnecessarily) OCaml encodes the build flags into the hashes of one
> of the base modules.  I forget the exact details but there's an email
> on this list about it somewhere.  Anyway the upshot is that if the
> build flags change at all, as they obviously do in Fedora vs RHEL,
> then the hashes change.  We ought to fix this upstream ...
>
> Anyway I was going to say why not order the packages by their original
> build date in Koji?



For one, this doesn’t actually mean that it’ll happen in order, since if we
are merging a side tag with a low-level package getting bootstrapped, it’s
possible that the packages getting tagged back to Rawhide are not in the
correct order.

For another, the only way then to ensure that they all build in the same
order is to serialize them. That adds so much latency to the build process
that after a few weeks, rebuilds would lag by days or more.

So, we cheat. We take a couple shortcuts that results in about 99% of our
builds working properly and then we deal with the remainder. Usually this
is in the form of build failures which are easy to spot. Sometimes we have
issues like this one where the build succeeds but the built package is
wrong, which we really only find out about when someone tells us.

>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Richard W.M. Jones]

On Fri, Oct 06, 2023 at 11:16:17AM -0400, Stephen Gallagher wrote:
> So, as we all know, build ordering is hard (and, despite intuitive
> belief, not actually deterministic).
> 
> ELN actually "cheats" somewhat when we do our builds. When we process
> a batch of builds (triggered by a set of tag events that come in all
> at the same time, such as when a side-tag is merged), we create a new
> ELN side-tag, tag all of the new *Rawhide* builds into this side tag,
> then trigger a rebuild of all of those builds for ELN. The result here
> is that we use the Fedora build in the buildroot to avoid
> bootstrapping issues. Now, there are some special-case packages for
> which we do *NOT* automatically tag in the Fedora builds because they
> have known incompatibilities. All OCAML packages fall into this
> category, since we discovered about 9 months ago that we absolutely
> cannot mix Fedora's OCAML builds with ELN's (I don't recall the exact
> reason).

It's probably because the build flags are different and
(unnecessarily) OCaml encodes the build flags into the hashes of one
of the base modules.  I forget the exact details but there's an email
on this list about it somewhere.  Anyway the upshot is that if the
build flags change at all, as they obviously do in Fedora vs RHEL,
then the hashes change.  We ought to fix this upstream ...

Anyway I was going to say why not order the packages by their original
build date in Koji?

Rich.

> Our other "cheat" when we rebuild a batch is that we automatically do
> rebuilds at least once for any failure, to account for things like
> build ordering and test flakes. In the case you're describing,
> unforunately, we had a situation where 1) it was OCAML, and therefore
> the Fedora packages weren't in the buildroot and 2) ocaml built
> successfully against the older version of the macros. If the
> BuildRequires: had been in play there, it would have failed, the batch
> would have finished building whatever else was able to succeed (such
> as the macros) and then the second pass would have succeeded.
> 
> I'm sorry you got hit by this, Richard. It's an unfortunate confluence
> of limitations in our rebuild approach.
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Stephen Gallagher]

On Fri, Oct 6, 2023 at 7:38 AM Richard W.M. Jones  wrote:
>
>
> On Fri, Oct 06, 2023 at 12:23:09PM +0100, Richard W.M. Jones wrote:
> > On Fri, Oct 06, 2023 at 10:19:22AM +0100, Richard W.M. Jones wrote:
> > > On Fri, Oct 06, 2023 at 08:48:52AM +0100, Richard W.M. Jones wrote:
> > > > On Thu, Oct 05, 2023 at 10:03:59AM +0100, Richard W.M. Jones wrote:
> > > > > Over the next few days I'm going to rebuild all OCaml packages in
> > > > > Rawhide for OCaml 5.1, plus some non-OCaml packages that have OCaml
> > > > > bindings.
> > > > >
> > > > > OCaml 5.1 is basically a small point release, but it does add back
> > > > > native code support for riscv64 and s390x.  The only remaining
> > > > > architecture that hasn't been updated for OCaml 5 (and is therefore
> > > > > still using the bytecode interpreter) is ppc64le.
> > > >
> > > > I think the builds are complete, except one which is running now.
> > > >
> > > > Only swig failed to build but that appears to be a general FTBFS bug:
> > > >   https://bugzilla.redhat.com/show_bug.cgi?id=2242372
> > > >
> > > > I'll submit an update later today once I've done a few more checks.
> > >
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-f0270d1637
> >
> > ELN builds are failing because they need to be done in build order,
> > not in random or alphabetical order or whatever ELN uses, so that's a
> > thing ...
>
> Actually it's worse.  Because this build of ocaml:
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=107128822
>
> was built before ocaml-srpm-macros was updated, it is being built
> wrongly.  The s390x build.log contains "--disable-native-compiler" but
> it should be the opposite since native compilation is now supported.
>
> Now partly this is a mistake in the spec file which should BR
> ocaml-srpm-macros >= 9 (I'll fix that shortly), but partly this is
> caused by the wrong build ordering of ELN.
>

So, as we all know, build ordering is hard (and, despite intuitive
belief, not actually deterministic).

ELN actually "cheats" somewhat when we do our builds. When we process
a batch of builds (triggered by a set of tag events that come in all
at the same time, such as when a side-tag is merged), we create a new
ELN side-tag, tag all of the new *Rawhide* builds into this side tag,
then trigger a rebuild of all of those builds for ELN. The result here
is that we use the Fedora build in the buildroot to avoid
bootstrapping issues. Now, there are some special-case packages for
which we do *NOT* automatically tag in the Fedora builds because they
have known incompatibilities. All OCAML packages fall into this
category, since we discovered about 9 months ago that we absolutely
cannot mix Fedora's OCAML builds with ELN's (I don't recall the exact
reason).

Our other "cheat" when we rebuild a batch is that we automatically do
rebuilds at least once for any failure, to account for things like
build ordering and test flakes. In the case you're describing,
unforunately, we had a situation where 1) it was OCAML, and therefore
the Fedora packages weren't in the buildroot and 2) ocaml built
successfully against the older version of the macros. If the
BuildRequires: had been in play there, it would have failed, the batch
would have finished building whatever else was able to succeed (such
as the macros) and then the second pass would have succeeded.

I'm sorry you got hit by this, Richard. It's an unfortunate confluence
of limitations in our rebuild approach.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: ELN build order (was: Re: OCaml 5.1 rebuild)

[Stephen Smoogen]

On Fri, 6 Oct 2023 at 07:38, Richard W.M. Jones  wrote:

>
> On Fri, Oct 06, 2023 at 12:23:09PM +0100, Richard W.M. Jones wrote:
> > On Fri, Oct 06, 2023 at 10:19:22AM +0100, Richard W.M. Jones wrote:
> > > On Fri, Oct 06, 2023 at 08:48:52AM +0100, Richard W.M. Jones wrote:
> > > > On Thu, Oct 05, 2023 at 10:03:59AM +0100, Richard W.M. Jones wrote:
> > > > > Over the next few days I'm going to rebuild all OCaml packages in
> > > > > Rawhide for OCaml 5.1, plus some non-OCaml packages that have OCaml
> > > > > bindings.
> > > > >
> > > > > OCaml 5.1 is basically a small point release, but it does add back
> > > > > native code support for riscv64 and s390x.  The only remaining
> > > > > architecture that hasn't been updated for OCaml 5 (and is therefore
> > > > > still using the bytecode interpreter) is ppc64le.
> > > >
> > > > I think the builds are complete, except one which is running now.
> > > >
> > > > Only swig failed to build but that appears to be a general FTBFS bug:
> > > >   https://bugzilla.redhat.com/show_bug.cgi?id=2242372
> > > >
> > > > I'll submit an update later today once I've done a few more checks.
> > >
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2023-f0270d1637
> >
> > ELN builds are failing because they need to be done in build order,
> > not in random or alphabetical order or whatever ELN uses, so that's a
> > thing ...
>
> Actually it's worse.  Because this build of ocaml:
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=107128822
>
> was built before ocaml-srpm-macros was updated, it is being built
> wrongly.  The s390x build.log contains "--disable-native-compiler" but
> it should be the opposite since native compilation is now supported.
>
> Now partly this is a mistake in the spec file which should BR
> ocaml-srpm-macros >= 9 (I'll fix that shortly), but partly this is
> caused by the wrong build ordering of ELN.
>

If they are using the standard mass rebuild method, it is just feeding
everything in alphabetical order, but it may still get 'reordered' somewhat
by koji. I believe the main problem is that no one has ever hacked any sort
of 'build recipe' to koji for it to 'know' that things need specific build
orders or bootstrap instructions. Instead that has been left to the
'maintainers' of the packages because in general most of them required a
lot of specific knowledge of the code to do right (or various maintainers
do not want ANYONE ELSE to touch their package.) [Recipes got pulled into
modularity but you also got all the rest of modularity also which was not
what many maintainers wanted either]


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

ELN build order (was: Re: OCaml 5.1 rebuild)

[Richard W.M. Jones]

On Fri, Oct 06, 2023 at 12:23:09PM +0100, Richard W.M. Jones wrote:
> On Fri, Oct 06, 2023 at 10:19:22AM +0100, Richard W.M. Jones wrote:
> > On Fri, Oct 06, 2023 at 08:48:52AM +0100, Richard W.M. Jones wrote:
> > > On Thu, Oct 05, 2023 at 10:03:59AM +0100, Richard W.M. Jones wrote:
> > > > Over the next few days I'm going to rebuild all OCaml packages in
> > > > Rawhide for OCaml 5.1, plus some non-OCaml packages that have OCaml
> > > > bindings.
> > > > 
> > > > OCaml 5.1 is basically a small point release, but it does add back
> > > > native code support for riscv64 and s390x.  The only remaining
> > > > architecture that hasn't been updated for OCaml 5 (and is therefore
> > > > still using the bytecode interpreter) is ppc64le.
> > > 
> > > I think the builds are complete, except one which is running now.
> > > 
> > > Only swig failed to build but that appears to be a general FTBFS bug:
> > >   https://bugzilla.redhat.com/show_bug.cgi?id=2242372
> > > 
> > > I'll submit an update later today once I've done a few more checks.
> > 
> > https://bodhi.fedoraproject.org/updates/FEDORA-2023-f0270d1637
> 
> ELN builds are failing because they need to be done in build order,
> not in random or alphabetical order or whatever ELN uses, so that's a
> thing ...

Actually it's worse.  Because this build of ocaml:

https://koji.fedoraproject.org/koji/taskinfo?taskID=107128822

was built before ocaml-srpm-macros was updated, it is being built
wrongly.  The s390x build.log contains "--disable-native-compiler" but
it should be the opposite since native compilation is now supported.

Now partly this is a mistake in the spec file which should BR
ocaml-srpm-macros >= 9 (I'll fix that shortly), but partly this is
caused by the wrong build ordering of ELN.

Rich.

> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: ELN build order

[Vít Ondruch]

Thinking about this, would it be possible to disable automatic ELN 
rebuild after side-tag merge? Such packages could be flagged until they 
are build manually. Of course it would be nice if automation did it 
properly, but the manual work would be much better then broken ELN.



Vít


Dne 11. 01. 21 v 11:54 Vít Ondruch napsal(a):


Can the ELN be fixed to stop doing rebuild like this? Of course the 
issue is not in rubygem-thin but the issue is that once the Ruby side 
tag was merged, ELN did not bothered with build order and now there is 
rubygem-eventmachine which was built against Ruby 2.7 where is should 
have been build against Ruby 3.0. I can't rebuild it, because that 
would require release bump in Fedora (I don't want to do that, because 
Fedora maintainers should not be bothered with ELN, right?).


I think that ELN should try to follow the Rawhide build order. If it 
can't do that, then it should wait for manual intervention.


I think that ELN should have done gating or scratch build and check if 
the dependencies are the same as in Rawhide or something.


I think that ELN should check the build result for dependency issue 
and stop blindly submitting builds.




Vít




 Přeposlaná zpráva 
Předmět: 	bpeck/jenkins-continuous-infra.apps.ci.centos.org's 
rubygem-thin-1.7.2-16.eln108 failed to build

Datum:  Mon, 11 Jan 2021 08:49:37 + (GMT)
Od: notificati...@fedoraproject.org
Komu:   vondr...@redhat.com



Notification time stamped 2021-01-11 08:49:35 UTC

bpeck/jenkins-continuous-infra.apps.ci.centos.org's 
rubygem-thin-1.7.2-16.eln108 failed to build

http://koji.fedoraproject.org/koji/buildinfo?buildID=128

--
You received this message due to your preference settings at 
https://apps.fedoraproject.org/notifications/vondruch.id.fedoraproject.org/email/50912


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Fabio Valentini]

On Mon, Jan 25, 2021 at 3:48 PM Vít Ondruch  wrote:
>

(snip)

>> I think that ELN should try to follow the Rawhide build order. If it can't 
>> do that, then it should wait for manual intervention.
>
>
> This is on our todo list, but we have been balancing a lot of things at the 
> same time and haven't been able to get to it yet. Believe me, we want this 
> fixed ASAP because it causes us (the ELN maintainers) a lot of problems.
>
>
> Any update? Over the weekend, there was done over 238 failed build attempts. 
> This is certainly waste of resources.
>
>
> Can the offending build be completely deleted from Koji for example? I know 
> this is generally not a good idea, but since ELN is not targeting end users, 
> it won't hurt IMO.

This happened again with the boost 1.75 rebuilds. Now ELN buildroot is
broken because rpm-build is not installable :)
So *PLEASE* get this build order stuff sorted. It's massively annoying
for packagers and waseful of already scarce computing resources ...

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Vít Ondruch]

Dne 25. 01. 21 v 10:20 Vít Ondruch napsal(a):



Dne 11. 01. 21 v 15:15 Stephen Gallagher napsal(a):



On Mon, Jan 11, 2021 at 5:55 AM Vít Ondruch > wrote:


Can the ELN be fixed to stop doing rebuild like this? Of course
the issue is not in rubygem-thin but the issue is that once the
Ruby side tag was merged, ELN did not bothered with build order
and now there is rubygem-eventmachine which was built against
Ruby 2.7 where is should have been build against Ruby 3.0. I
can't rebuild it, because that would require release bump in
Fedora (I don't want to do that, because Fedora maintainers
should not be bothered with ELN, right?).

I think that ELN should try to follow the Rawhide build order. If
it can't do that, then it should wait for manual intervention.


This is on our todo list, but we have been balancing a lot of things 
at the same time and haven't been able to get to it yet. Believe me, 
we want this fixed ASAP because it causes us (the ELN maintainers) a 
lot of problems.



Any update? Over the weekend, there was done over 238 failed build 
attempts. This is certainly waste of resources.




Can the offending build be completely deleted from Koji for example? I 
know this is generally not a good idea, but since ELN is not targeting 
end users, it won't hurt IMO.



Vít





Vít



I think that ELN should have done gating or scratch build and
check if the dependencies are the same as in Rawhide or something.


I'm not sure where or when in the process you are suggesting this 
would happen. ELN gates the compose, but not the builds.


I think that ELN should check the build result for dependency
issue and stop blindly submitting builds.

Again, I'm not sure what you're asking for, here.


___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of 
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List 
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


OpenPGP_0x0CE09EE79917B87C.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Vít Ondruch]

Dne 11. 01. 21 v 15:15 Stephen Gallagher napsal(a):



On Mon, Jan 11, 2021 at 5:55 AM Vít Ondruch > wrote:


Can the ELN be fixed to stop doing rebuild like this? Of course
the issue is not in rubygem-thin but the issue is that once the
Ruby side tag was merged, ELN did not bothered with build order
and now there is rubygem-eventmachine which was built against Ruby
2.7 where is should have been build against Ruby 3.0. I can't
rebuild it, because that would require release bump in Fedora (I
don't want to do that, because Fedora maintainers should not be
bothered with ELN, right?).

I think that ELN should try to follow the Rawhide build order. If
it can't do that, then it should wait for manual intervention.


This is on our todo list, but we have been balancing a lot of things 
at the same time and haven't been able to get to it yet. Believe me, 
we want this fixed ASAP because it causes us (the ELN maintainers) a 
lot of problems.



Any update? Over the weekend, there was done over 238 failed build 
attempts. This is certainly waste of resources.



Vít



I think that ELN should have done gating or scratch build and
check if the dependencies are the same as in Rawhide or something.


I'm not sure where or when in the process you are suggesting this 
would happen. ELN gates the compose, but not the builds.


I think that ELN should check the build result for dependency
issue and stop blindly submitting builds.

Again, I'm not sure what you're asking for, here.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


OpenPGP_0x0CE09EE79917B87C.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Miro Hrončok]

On 14. 01. 21 11:15, Zbigniew Jędrzejewski-Szmek wrote:

On Mon, Jan 11, 2021 at 09:22:26PM +0100, Fabio Valentini wrote:

On Mon, Jan 11, 2021 at 3:57 PM Vít Ondruch  wrote:


Also, rubygem-eventmachine should be installable after rebuild. But certainly, 
there might happen race conditions a it happened this time.


This reminds me - over a year ago, I triggered discussion about making
the dist.rpmdeplint test blocking for all packages, which would solve
this exact problem. However, taskotron was retired shortly after, and
Fedora CI has only been catching up lately. Is there an installability
check in Fedora CI now? Can we make it so that if packages contained
in an update do not install correctly the update fails gating checks?


Yes, there is:
see e.g. https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f926ffe2c:

fedora-ci.koji-build.tier0.functional
fedora-ci.koji-build.installability.functional
fedora-ci.koji-build.rpmdeplint.functional
fedora-ci.koji-build.compose-ci.static-analysis
fedora-ci.koji-build.rpminspect.static-analysis

fedora-ci.koji-build.installability.functional would seem like the right
thing, but it currently is rather crude: systemd doesn't pass because it
has mutually conflicting subpackages (on purpose...). There is no obvious
opt-out mechanism, excepting disabling the test as a whole. Hopefully this
can be improved to the point where it can be made blocking for all updates.


It also report conflicts with the previous version of self:

https://bodhi.fedoraproject.org/updates/FEDORA-2021-0b0b2c54db

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jan 11, 2021 at 09:22:26PM +0100, Fabio Valentini wrote:
> On Mon, Jan 11, 2021 at 3:57 PM Vít Ondruch  wrote:
> >
> > Also, rubygem-eventmachine should be installable after rebuild. But 
> > certainly, there might happen race conditions a it happened this time.
>
> This reminds me - over a year ago, I triggered discussion about making
> the dist.rpmdeplint test blocking for all packages, which would solve
> this exact problem. However, taskotron was retired shortly after, and
> Fedora CI has only been catching up lately. Is there an installability
> check in Fedora CI now? Can we make it so that if packages contained
> in an update do not install correctly the update fails gating checks?

Yes, there is:
see e.g. https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f926ffe2c:

fedora-ci.koji-build.tier0.functional
fedora-ci.koji-build.installability.functional
fedora-ci.koji-build.rpmdeplint.functional
fedora-ci.koji-build.compose-ci.static-analysis
fedora-ci.koji-build.rpminspect.static-analysis

fedora-ci.koji-build.installability.functional would seem like the right
thing, but it currently is rather crude: systemd doesn't pass because it
has mutually conflicting subpackages (on purpose...). There is no obvious
opt-out mechanism, excepting disabling the test as a whole. Hopefully this
can be improved to the point where it can be made blocking for all updates.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Miroslav Vadkerti]

Actually that would not help in this situation, I was talking about Fedora,
ELN builds are not planned to be gated ..

I stand corrected, thanks @Michal Srb 

/M

On Thu, Jan 14, 2021 at 9:42 AM Miroslav Vadkerti 
wrote:

> Hi,
>
> can we share the timeline for making rpmdeplint and installability gating
> pls? Not sure where we have it.
>
> Thank you!,
> /M
>
> On Mon, Jan 11, 2021 at 9:23 PM Fabio Valentini 
> wrote:
>
>> On Mon, Jan 11, 2021 at 3:57 PM Vít Ondruch  wrote:
>> >
>> > Also, rubygem-eventmachine should be installable after rebuild. But
>> certainly, there might happen race conditions a it happened this time.
>>
>> This reminds me - over a year ago, I triggered discussion about making
>> the dist.rpmdeplint test blocking for all packages, which would solve
>> this exact problem. However, taskotron was retired shortly after, and
>> Fedora CI has only been catching up lately. Is there an installability
>> check in Fedora CI now? Can we make it so that if packages contained
>> in an update do not install correctly the update fails gating checks?
>>
>> Fabio
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>>
>
>
> --
> Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI
> IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
> TPB-C 2C215 :: Mobile +420 773 944 252
> Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
>
>
>

-- 
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile +420 773 944 252
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Miroslav Vadkerti]

Hi,

can we share the timeline for making rpmdeplint and installability gating
pls? Not sure where we have it.

Thank you!,
/M

On Mon, Jan 11, 2021 at 9:23 PM Fabio Valentini 
wrote:

> On Mon, Jan 11, 2021 at 3:57 PM Vít Ondruch  wrote:
> >
> > Also, rubygem-eventmachine should be installable after rebuild. But
> certainly, there might happen race conditions a it happened this time.
>
> This reminds me - over a year ago, I triggered discussion about making
> the dist.rpmdeplint test blocking for all packages, which would solve
> this exact problem. However, taskotron was retired shortly after, and
> Fedora CI has only been catching up lately. Is there an installability
> check in Fedora CI now? Can we make it so that if packages contained
> in an update do not install correctly the update fails gating checks?
>
> Fabio
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>


-- 
Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95
TPB-C 2C215 :: Mobile +420 773 944 252
Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Fabio Valentini]

On Mon, Jan 11, 2021 at 3:57 PM Vít Ondruch  wrote:
>
> Also, rubygem-eventmachine should be installable after rebuild. But 
> certainly, there might happen race conditions a it happened this time.

This reminds me - over a year ago, I triggered discussion about making
the dist.rpmdeplint test blocking for all packages, which would solve
this exact problem. However, taskotron was retired shortly after, and
Fedora CI has only been catching up lately. Is there an installability
check in Fedora CI now? Can we make it so that if packages contained
in an update do not install correctly the update fails gating checks?

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Vít Ondruch]

Dne 11. 01. 21 v 15:15 Stephen Gallagher napsal(a):



On Mon, Jan 11, 2021 at 5:55 AM Vít Ondruch > wrote:


Can the ELN be fixed to stop doing rebuild like this? Of course
the issue is not in rubygem-thin but the issue is that once the
Ruby side tag was merged, ELN did not bothered with build order
and now there is rubygem-eventmachine which was built against Ruby
2.7 where is should have been build against Ruby 3.0. I can't
rebuild it, because that would require release bump in Fedora (I
don't want to do that, because Fedora maintainers should not be
bothered with ELN, right?).

I think that ELN should try to follow the Rawhide build order. If
it can't do that, then it should wait for manual intervention.


This is on our todo list, but we have been balancing a lot of things 
at the same time and haven't been able to get to it yet. Believe me, 
we want this fixed ASAP because it causes us (the ELN maintainers) a 
lot of problems.


I think that ELN should have done gating or scratch build and
check if the dependencies are the same as in Rawhide or something.


I'm not sure where or when in the process you are suggesting this 
would happen. ELN gates the compose, but not the builds.



When ELN built rubygem-eventmachine, it should have make sure that the 
dependencies are as expected. If rubygem-eventmachine depends in Rawhide 
on libruby.so.3.0()(64bit), then the dependency on 
libruby.so.2.7()(64bit) should trigger alarm.


Also, rubygem-eventmachine should be installable after rebuild. But 
certainly, there might happen race conditions a it happened this time.




I think that ELN should check the build result for dependency
issue and stop blindly submitting builds.

Again, I'm not sure what you're asking for, here.



For a human, single rebuild and check of log for rubygem-thin would 
reveal that something is wrong with rubygem-eventmachine. This could be 
detected from root.log and flagged. More builds are not needed.



Vít



OpenPGP_0x0CE09EE79917B87C.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: ELN build order

[Stephen Gallagher]

On Mon, Jan 11, 2021 at 5:55 AM Vít Ondruch  wrote:

> Can the ELN be fixed to stop doing rebuild like this? Of course the issue
> is not in rubygem-thin but the issue is that once the Ruby side tag was
> merged, ELN did not bothered with build order and now there is
> rubygem-eventmachine which was built against Ruby 2.7 where is should have
> been build against Ruby 3.0. I can't rebuild it, because that would require
> release bump in Fedora (I don't want to do that, because Fedora maintainers
> should not be bothered with ELN, right?).
>
> I think that ELN should try to follow the Rawhide build order. If it can't
> do that, then it should wait for manual intervention.
>

This is on our todo list, but we have been balancing a lot of things at the
same time and haven't been able to get to it yet. Believe me, we want this
fixed ASAP because it causes us (the ELN maintainers) a lot of problems.


> I think that ELN should have done gating or scratch build and check if the
> dependencies are the same as in Rawhide or something.
>

I'm not sure where or when in the process you are suggesting this would
happen. ELN gates the compose, but not the builds.


> I think that ELN should check the build result for dependency issue and
> stop blindly submitting builds.
>
Again, I'm not sure what you're asking for, here.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

ELN build order

[Vít Ondruch]

Can the ELN be fixed to stop doing rebuild like this? Of course the 
issue is not in rubygem-thin but the issue is that once the Ruby side 
tag was merged, ELN did not bothered with build order and now there is 
rubygem-eventmachine which was built against Ruby 2.7 where is should 
have been build against Ruby 3.0. I can't rebuild it, because that would 
require release bump in Fedora (I don't want to do that, because Fedora 
maintainers should not be bothered with ELN, right?).


I think that ELN should try to follow the Rawhide build order. If it 
can't do that, then it should wait for manual intervention.


I think that ELN should have done gating or scratch build and check if 
the dependencies are the same as in Rawhide or something.


I think that ELN should check the build result for dependency issue and 
stop blindly submitting builds.




Vít




 Přeposlaná zpráva 
Předmět: 	bpeck/jenkins-continuous-infra.apps.ci.centos.org's 
rubygem-thin-1.7.2-16.eln108 failed to build

Datum:  Mon, 11 Jan 2021 08:49:37 + (GMT)
Od: notificati...@fedoraproject.org
Komu:   vondr...@redhat.com



Notification time stamped 2021-01-11 08:49:35 UTC

bpeck/jenkins-continuous-infra.apps.ci.centos.org's 
rubygem-thin-1.7.2-16.eln108 failed to build

http://koji.fedoraproject.org/koji/buildinfo?buildID=128

--
You received this message due to your preference settings at 
https://apps.fedoraproject.org/notifications/vondruch.id.fedoraproject.org/email/50912


OpenPGP_0x0CE09EE79917B87C.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org