Re: koji builder status
On 19. 12. 19 21:10, Kevin Fenzi wrote: On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote: Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest I'm not sure why that is. I would expect them to be busy more as well. I have the same experience with Python noarch builds. I have figured out that it is quite deterministic: When I need the build quickly and/or there are hundreds of slow-ish tests, I get arm. When I don't need the build quickly and/or there are no tests at all and it builds in seconds anywhere, I get x86. Based on that, I've determined that it is just a bias :D -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji builder status
On Fri, 20 Dec 2019 01:39:42 + Mat Booth wrote: > On Thu, 19 Dec 2019 at 21:13, Dan Horák wrote: > > > > On Thu, 19 Dec 2019 12:10:00 -0800 > > Kevin Fenzi wrote: > > > > > On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote: > > > > > > > > Side note: Considering that there are comparatively few armv7hl > > > > builders, I wonder why a rather big share of my noarch builds > > > > gets scheduled to run on them. Naively, I'd think that armv7hl > > > > should be busy doing builds for "archful" stuff since they're > > > > the slowest > > > > > > I'm not sure why that is. I would expect them to be busy more as > > > well. Do note that we are bringing some more hardware on line > > > after the new year, which should give us more armv7 builders. > > > > > > > builders and there are the fewest of them, except for s390x. > > > > > > s390x now is a great deal faster. It's often the one that finishes > > > right after x86_64/i686. > > > > even ahead x86_64/i686 sometimes :-) > > > > This is great to hear! Until now s390x was the slowest arch for Java > by quite a wide margin; with the special Java 8 JIT package[1] > installed, even 32bit arm was quicker than s390x :-o because there is no JIT for s390x in OpenJDK 1.8.0, it has been added in some later version (OpenJDK 10?). So until this version is the new default Java, s390x will be slow for Java. Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji builder status
On Thu, 19 Dec 2019 at 21:13, Dan Horák wrote: > > On Thu, 19 Dec 2019 12:10:00 -0800 > Kevin Fenzi wrote: > > > On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote: > > > > > > Side note: Considering that there are comparatively few armv7hl > > > builders, I wonder why a rather big share of my noarch builds gets > > > scheduled to run on them. Naively, I'd think that armv7hl should be > > > busy doing builds for "archful" stuff since they're the slowest > > > > I'm not sure why that is. I would expect them to be busy more as > > well. Do note that we are bringing some more hardware on line after > > the new year, which should give us more armv7 builders. > > > > > builders and there are the fewest of them, except for s390x. > > > > s390x now is a great deal faster. It's often the one that finishes > > right after x86_64/i686. > > even ahead x86_64/i686 sometimes :-) > This is great to hear! Until now s390x was the slowest arch for Java by quite a wide margin; with the special Java 8 JIT package[1] installed, even 32bit arm was quicker than s390x :-o [1] "java-1.8.0-openjdk-aarch32" -- https://koji.fedoraproject.org/koji/packageinfo?packageID=22627 -- Mat Booth http://fedoraproject.org/get-fedora ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji builder status
On Thu, 19 Dec 2019 12:10:00 -0800 Kevin Fenzi wrote: > On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote: > > > > Side note: Considering that there are comparatively few armv7hl > > builders, I wonder why a rather big share of my noarch builds gets > > scheduled to run on them. Naively, I'd think that armv7hl should be > > busy doing builds for "archful" stuff since they're the slowest > > I'm not sure why that is. I would expect them to be busy more as > well. Do note that we are bringing some more hardware on line after > the new year, which should give us more armv7 builders. > > > builders and there are the fewest of them, except for s390x. > > s390x now is a great deal faster. It's often the one that finishes > right after x86_64/i686. even ahead x86_64/i686 sometimes :-) Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji builder status
On Sun, Dec 15, 2019 at 10:24:16AM +0100, Fabio Valentini wrote: > > Side note: Considering that there are comparatively few armv7hl > builders, I wonder why a rather big share of my noarch builds gets > scheduled to run on them. Naively, I'd think that armv7hl should be > busy doing builds for "archful" stuff since they're the slowest I'm not sure why that is. I would expect them to be busy more as well. Do note that we are bringing some more hardware on line after the new year, which should give us more armv7 builders. > builders and there are the fewest of them, except for s390x. s390x now is a great deal faster. It's often the one that finishes right after x86_64/i686. > Since most of the noarch stuff I do for fedora nowadays is Java > (*sigh*), these Java-on-arm builds are rather slow. Maybe that's bad > luck, or maybe the task scheduler hates me :) Yeah, not sure. We can try and investigate more. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: koji builder status
On Fri, Dec 13, 2019 at 7:22 PM Kevin Fenzi wrote: > > Greetings. Hi Kevin, > I just thought I would share a short status update on koji builders with > everyone. As many of you know, we like to keep the Fedora koji builders > on the current most recent stable Fedora release. This is to make sure > we have compatible rpm, etc to build rawhide and all our stable updates. > TLDR summary: All Fedora koji builders are now on Fedora 31 and are > using only python3 (the hub is still python2 however). That's great news, thanks for the update, and for keeping fedora infra running smoothly! :) > Due to a bug in the linux kernel on armv7, we never upgraded to Fedora > 30 when it was released, so builders stayed on Fedora 29 for the last > cycle. Happily that bug was tracked down and fixed and I have been > moving them to Fedora 31 for the past month or so. > > Normally it doesn't take to long to reinstall everything (Thanks > ansible!) but this time it's been a longer road due to: New aarch64 > hardware that needed racking/cabling/setup (many thanks to Smooge for > getting this done), firmware updates for most all hardware, RHEL8 > reinstalls for virthosts that run buildvm's, and many other items. > > We still have pending: > > * Some more aarch64 hardware to bring online early in the new year. > * A bug around armv7 buildvm's with more than 24GB memory. While thats > being investigated all the buildvm-armv7 instances have just 24GB > memory. > * A bug around power9 hosts, causing our cloud/container image builds to > fail. This was thought fixed, but doesn't seem to be yet. > * Daily db dumps still cause slowness, likely in the new year we will > schedule an outage and patition the problem tables. > A few stats: > > 155 total enabled builders > 47 x86_64 builders > 32 ppc64le builders > 32 aarch64 builders > 27 armhfp builders > 24 s390x builders Side note: Considering that there are comparatively few armv7hl builders, I wonder why a rather big share of my noarch builds gets scheduled to run on them. Naively, I'd think that armv7hl should be busy doing builds for "archful" stuff since they're the slowest builders and there are the fewest of them, except for s390x. Since most of the noarch stuff I do for fedora nowadays is Java (*sigh*), these Java-on-arm builds are rather slow. Maybe that's bad luck, or maybe the task scheduler hates me :) Fabio > Around 10,000 builds a month. > Close to 1.1 million builds since fc6 days > > Happy building. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
koji builder status
Greetings. I just thought I would share a short status update on koji builders with everyone. As many of you know, we like to keep the Fedora koji builders on the current most recent stable Fedora release. This is to make sure we have compatible rpm, etc to build rawhide and all our stable updates. TLDR summary: All Fedora koji builders are now on Fedora 31 and are using only python3 (the hub is still python2 however). Due to a bug in the linux kernel on armv7, we never upgraded to Fedora 30 when it was released, so builders stayed on Fedora 29 for the last cycle. Happily that bug was tracked down and fixed and I have been moving them to Fedora 31 for the past month or so. Normally it doesn't take to long to reinstall everything (Thanks ansible!) but this time it's been a longer road due to: New aarch64 hardware that needed racking/cabling/setup (many thanks to Smooge for getting this done), firmware updates for most all hardware, RHEL8 reinstalls for virthosts that run buildvm's, and many other items. We still have pending: * Some more aarch64 hardware to bring online early in the new year. * A bug around armv7 buildvm's with more than 24GB memory. While thats being investigated all the buildvm-armv7 instances have just 24GB memory. * A bug around power9 hosts, causing our cloud/container image builds to fail. This was thought fixed, but doesn't seem to be yet. * Daily db dumps still cause slowness, likely in the new year we will schedule an outage and patition the problem tables. A few stats: 155 total enabled builders 47 x86_64 builders 32 ppc64le builders 32 aarch64 builders 27 armhfp builders 24 s390x builders Around 10,000 builds a month. Close to 1.1 million builds since fc6 days Happy building. signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
