Re: KOJI armv7hl builders out of resources

2021-08-04 Thread Michal Schorm 
On Wed, Aug 4, 2021 at 6:31 PM Kevin Fenzi  wrote:
> Hopefully it fixes the community-mysql issue?

Yes, I got a successful build on the first try without touching the
Rawhide code.
  https://koji.fedoraproject.org/koji/taskinfo?taskID=73279974

Thank you very much for your help, Kevin !

Michal

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--

On Wed, Aug 4, 2021 at 6:31 PM Kevin Fenzi  wrote:
>
> On Tue, Aug 03, 2021 at 08:50:24AM -0700, Kevin Fenzi wrote:
> >
> > Yes, stupidly they updated to the normal kernel before the last reboot.
> > So, they only see 3GB of their 40GB memory. :(
> >
> > I'll fix and reboot them today. I have no idea why this wasn't seen
> > before now. I guess it shows that they are not really memory bound
> > normally. :(
>
> Just to note that I fixed and rebooted all the armv7 builders yesterday.
>
> Sadly, this seems to have brought back
> https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> for python builds again.
>
> Hopefully it fixes the community-mysql issue?
>
> kevin
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it: 
> https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-04 Thread Kevin Fenzi 
On Tue, Aug 03, 2021 at 08:50:24AM -0700, Kevin Fenzi wrote:
> 
> Yes, stupidly they updated to the normal kernel before the last reboot. 
> So, they only see 3GB of their 40GB memory. :( 
> 
> I'll fix and reboot them today. I have no idea why this wasn't seen
> before now. I guess it shows that they are not really memory bound
> normally. :( 

Just to note that I fixed and rebooted all the armv7 builders yesterday. 

Sadly, this seems to have brought back
https://bugzilla.redhat.com/show_bug.cgi?id=1920183
for python builds again. 

Hopefully it fixes the community-mysql issue?

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-03 Thread Kevin Fenzi 
On Tue, Aug 03, 2021 at 03:24:18PM +0200, Michal Schorm wrote:
> Thank you for the suggestions.
> I'll test reducing the debuginfo.
> 
> However I still believe something else is in play, as such errors are
> normally very rare.
> The package has already been built without issues; it was the F35 mass
> rebuild which only bumped release, after which I noticed the FTBFS.
> 
> I also maintain a mysql module based on the same code. That's a lot of
> builds already, so I would likely spot build issues, if they would
> occur often.
> 
> Is there a way to check how much resources the KOJI build actually consumed ?

The only recent change in builders was moving them to f34. 
That was in early April. Since then they have only had normal f34
updates. 

You can see the hardware stats of the builder in the hw_info.log file
thats next to build.log/root.log in koji. So for example, the
community-mysql mass rebuild failure has: 

https://kojipkgs.fedoraproject.org//work/tasks/6534/72336534/hw_info.log

sigh. 

Yes, stupidly they updated to the normal kernel before the last reboot. 
So, they only see 3GB of their 40GB memory. :( 

I'll fix and reboot them today. I have no idea why this wasn't seen
before now. I guess it shows that they are not really memory bound
normally. :( 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-03 Thread Michal Schorm 
Thank you for the suggestions.
I'll test reducing the debuginfo.

However I still believe something else is in play, as such errors are
normally very rare.
The package has already been built without issues; it was the F35 mass
rebuild which only bumped release, after which I noticed the FTBFS.

I also maintain a mysql module based on the same code. That's a lot of
builds already, so I would likely spot build issues, if they would
occur often.

Is there a way to check how much resources the KOJI build actually consumed ?

Michal

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--

On Tue, Aug 3, 2021 at 11:44 AM Vitaly Zaitsev via devel
 wrote:
>
> On 03/08/2021 11:31, Lukas Javorsky wrote:
> > And is there any suggested way how to handle this situation?
>
> Special hacks for ARMv7:
>
> 1. Build in one thread.
> 2. Reduce debug info: -g -> -g1 or even -g0.
>
> > Some spec build macro or something?
>
> %ifarch %{arm}
> %global optflags %(echo %{optflags} | sed 's/-g /-g0 /')
> %endif
>
> --
> Sincerely,
>Vitaly Zaitsev (vit...@easycoding.org)
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it: 
> https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-03 Thread Vitaly Zaitsev via devel 

On 03/08/2021 11:31, Lukas Javorsky wrote:

And is there any suggested way how to handle this situation?


Special hacks for ARMv7:

1. Build in one thread.
2. Reduce debug info: -g -> -g1 or even -g0.


Some spec build macro or something?


%ifarch %{arm}
%global optflags %(echo %{optflags} | sed 's/-g /-g0 /')
%endif

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-03 Thread Lukas Javorsky 
> I suspect it's not the builder that's running out of resources but a
> single process exceeding the per process memory limit for 32 bit
> applications.

And is there any suggested way how to handle this situation?
Some spec build macro or something?

Thank you

On Tue, Aug 3, 2021 at 11:16 AM Peter Robinson  wrote:

> On Tue, Aug 3, 2021 at 7:09 AM Michal Schorm  wrote:
> >
> > Hello,
> >
> > recently, package 'community-mysql' started to FTBFS, because of lack
> > of resources on armv7hl builders in KOJI.
> >
> > Packages like 'mariadb' and 'community-mysql' are quite resource
> > hungry, both for disk and memory.
> >
> > Occasionally it happens that out of random reasons, some builder here
> > and there don't have enough resources. Usually because that particular
> > builder was under heavy load.
> >
> > However this time, the armv7hl builds lack the resources in 100%
> > builds for days in various times on various hosts.
> >
> > (1) Did the KOJI armv7hl builder's configuration change recently ?
> >
> > (2) Should I file an issue somewhere?
> > Generic https://pagure.io/releng/new_issue ?
> >
> > First spotted: F35 mass rebuild.
> > Reproducible: 100%, on all F33..Rawhide
>
> I suspect it's not the builder that's running out of resources but a
> single process exceeding the per process memory limit for 32 bit
> applications.
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>


-- 
S pozdravom/ Best regards

Lukáš Javorský

Associate Software Engineer, Core service - Databases

Red Hat 

Purkyňova 115 (TPB-C)

612 00 Brno - Královo Pole

ljavo...@redhat.com

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: KOJI armv7hl builders out of resources

2021-08-03 Thread Peter Robinson 
On Tue, Aug 3, 2021 at 7:09 AM Michal Schorm  wrote:
>
> Hello,
>
> recently, package 'community-mysql' started to FTBFS, because of lack
> of resources on armv7hl builders in KOJI.
>
> Packages like 'mariadb' and 'community-mysql' are quite resource
> hungry, both for disk and memory.
>
> Occasionally it happens that out of random reasons, some builder here
> and there don't have enough resources. Usually because that particular
> builder was under heavy load.
>
> However this time, the armv7hl builds lack the resources in 100%
> builds for days in various times on various hosts.
>
> (1) Did the KOJI armv7hl builder's configuration change recently ?
>
> (2) Should I file an issue somewhere?
> Generic https://pagure.io/releng/new_issue ?
>
> First spotted: F35 mass rebuild.
> Reproducible: 100%, on all F33..Rawhide

I suspect it's not the builder that's running out of resources but a
single process exceeding the per process memory limit for 32 bit
applications.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure