Re: Specify koji build machine mem req via. spec file

2023-10-06 Thread Adam Williamson 
On Fri, 2023-10-06 at 14:17 -0400, Stephen Gallagher wrote:
> On Thu, Oct 5, 2023 at 2:04 AM Adam Williamson
>  wrote:
> > 
> > On Wed, 2023-10-04 at 12:19 +0200, Kalev Lember wrote:
> > > On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky  
> > > wrote:
> > > 
> > > > Hello guys,
> > > > 
> > > > Is there's a way how to set requested amount of ram for koji builders?
> > > > 
> > > > I'd like to use it as Firefox builds fail recently due low memory, like
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=2241690
> > > 
> > > 
> > > I looked at the first linked build log in the ticket and it looks like the
> > > build failed in the debuginfo extraction step. In that log, find-debuginfo
> > > is called with -j3, and it could be that it ran out of memory because it
> > > was trying to do 3 debuginfo extractions in parallel. You could try to
> > > stick something like this in the spec file to tune it down:
> > > 
> > > # Require 32 GB of RAM per CPU core for debuginfo processing
> > > %global _find_debuginfo_opts %limit_build -m 32768
> > 
> > Thanks for the idea, Kalev. I went ahead and applied this, and builds
> > for F38 and F39 succeeded. Let's hope it really helps and this wasn't
> > just luck!
> 
> Looks like probably not, since the follow-up ELN build *also*
> succeeded for the first time in quite a while!

Unfortunately I still got several failed builds today while trying to
do builds with the fixes for the .desktop file issues :( Most recent
was a case of gdb-add-index getting killed, as Kevin says he found
happens with 24G of RAM:

https://bugzilla.redhat.com/show_bug.cgi?id=2241690#c7
https://koji.fedoraproject.org/koji/taskinfo?taskID=107145177

this was a build on buildvm-x86-11.iad2.fedoraproject.org . I guess
I'll try fiddling with some of the other stuff discussed in this thread
and firing more builds. :(
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org
https://www.happyassassin.net



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-06 Thread Stephen Gallagher 
On Thu, Oct 5, 2023 at 2:04 AM Adam Williamson
 wrote:
>
> On Wed, 2023-10-04 at 12:19 +0200, Kalev Lember wrote:
> > On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky  wrote:
> >
> > > Hello guys,
> > >
> > > Is there's a way how to set requested amount of ram for koji builders?
> > >
> > > I'd like to use it as Firefox builds fail recently due low memory, like
> > > https://bugzilla.redhat.com/show_bug.cgi?id=2241690
> >
> >
> > I looked at the first linked build log in the ticket and it looks like the
> > build failed in the debuginfo extraction step. In that log, find-debuginfo
> > is called with -j3, and it could be that it ran out of memory because it
> > was trying to do 3 debuginfo extractions in parallel. You could try to
> > stick something like this in the spec file to tune it down:
> >
> > # Require 32 GB of RAM per CPU core for debuginfo processing
> > %global _find_debuginfo_opts %limit_build -m 32768
>
> Thanks for the idea, Kalev. I went ahead and applied this, and builds
> for F38 and F39 succeeded. Let's hope it really helps and this wasn't
> just luck!

Looks like probably not, since the follow-up ELN build *also*
succeeded for the first time in quite a while!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-06 Thread Tulio Magno Quites Machado Filho 
Julian Sikorski  writes:

> Does a hammer smaller than -g1 also exist for issues like:
>
> relocation truncated to fit: R_X86_64_32 against `.debug_info'
> https://kojipkgs.fedoraproject.org//work/tasks/1615/107091615/build.log

This is saying that mame is too large to fit in the default code model
(-mcmodel=small).

You may need to build with -mcmodel=medium or -mcmodel=large in order to
get relocations that can deal with this kind of large binaries.

Linking agains shared libraries could also help in this case, but it may
be more difficult if upstream does not support it.

-- 
Tulio Magno
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-05 Thread Kalev Lember 
On Thu, Oct 5, 2023 at 2:04 PM Julian Sikorski  wrote:

> Does a hammer smaller than -g1 also exist for issues like:
>
> relocation truncated to fit: R_X86_64_32 against `.debug_info'
> https://kojipkgs.fedoraproject.org//work/tasks/1615/107091615/build.log
>
> Thanks for the info in advance.
>

Sorry, I don't know, that's something slightly different. Maybe open a
ticket against gcc and ask the tools team?

-- 
Kalev
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-05 Thread Julian Sikorski 

Am 04.10.23 um 12:19 schrieb Kalev Lember:
On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky > wrote:


Hello guys,

Is there's a way how to set requested amount of ram for koji builders?

I'd like to use it as Firefox builds fail recently due low memory, like
https://bugzilla.redhat.com/show_bug.cgi?id=2241690



I looked at the first linked build log in the ticket and it looks like 
the build failed in the debuginfo extraction step. In that log, 
find-debuginfo is called with -j3, and it could be that it ran out of 
memory because it was trying to do 3 debuginfo extractions in parallel. 
You could try to stick something like this in the spec file to tune it down:


# Require 32 GB of RAM per CPU core for debuginfo processing
%global _find_debuginfo_opts %limit_build -m 32768

... and see if it helps. With this, it should pass -j1 to find-debuginfo 
when the builder has 32 GB RAM, -j2 when it has 64 GB etc.


If that doesn't work, then there's also the option of using -g1 to 
reduce the binary sizes so they take less memory to process (but that 
makes backtraces much worse so I'd only leave it for emergencies if you 
can't get an important build out otherwise).



Does a hammer smaller than -g1 also exist for issues like:

relocation truncated to fit: R_X86_64_32 against `.debug_info'
https://kojipkgs.fedoraproject.org//work/tasks/1615/107091615/build.log

Thanks for the info in advance.

Best regards,
Julian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-05 Thread Panu Matilainen 

On 10/5/23 12:41, Kalev Lember wrote:
On Thu, Oct 5, 2023 at 10:07 AM Miroslav Suchý > wrote:


Dne 04. 10. 23 v 11:43 Martin Stransky napsal(a):
 > Hello guys,
 >
 > Is there's a way how to set requested amount of ram for koji
builders?
 >
 > I'd like to use it as Firefox builds fail recently due low
memory, like
 > https://bugzilla.redhat.com/show_bug.cgi?id=2241690

 >
 > Thanks,
 > Martin
 >
Related - rpm has tunables that allows you to tune the build
according the available resources. However memory is not there:

https://github.com/rpm-software-management/rpm/pull/2418


https://rpm-software-management.github.io/rpm/manual/buildprocess.html 



Oh, nice, I didn't know about that! From a quick look, there are 
actually two memory tunables, %_smp_tasksize_proc and %_smp_tasksize_thread.


Looks like the new thing would replace the constrain_build macro that 
firefox.spec is using:


Instead of current:

# Require 4 GB of RAM per CPU core
%constrain_build -m 4096

It would use:

# Require 4 GB of RAM per CPU core
%global _smp_tasksize_proc 4096

It doesn't look like a huge win from the packager's point of view, but I 
am sure it's better to use the rpm upstream limits instead of external 
macros such as 
constrain_build. 


Yep, memory is certainly considered, it was really the main driver of 
that change.


Compared to %constrain_build the big win is that memory is always 
considered, regardless of whether the packager asked for it or not. It 
would be a useful experiment to drop that tunable from the spec because 
it probably isn't necessary anymore (with rpm >= 4.18 that is).


The other win is that the native version also covers threads, which 
helps on 32bit systems in particular.


- Panu -
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-05 Thread Kalev Lember 
On Thu, Oct 5, 2023 at 10:07 AM Miroslav Suchý  wrote:

> Dne 04. 10. 23 v 11:43 Martin Stransky napsal(a):
> > Hello guys,
> >
> > Is there's a way how to set requested amount of ram for koji builders?
> >
> > I'd like to use it as Firefox builds fail recently due low memory, like
> > https://bugzilla.redhat.com/show_bug.cgi?id=2241690
> >
> > Thanks,
> > Martin
> >
> Related - rpm has tunables that allows you to tune the build according the
> available resources. However memory is not there:
>
> https://github.com/rpm-software-management/rpm/pull/2418
>
> https://rpm-software-management.github.io/rpm/manual/buildprocess.html


Oh, nice, I didn't know about that! From a quick look, there are actually
two memory tunables, %_smp_tasksize_proc and %_smp_tasksize_thread.

Looks like the new thing would replace the constrain_build macro that
firefox.spec is using:

Instead of current:

# Require 4 GB of RAM per CPU core
%constrain_build -m 4096

It would use:

# Require 4 GB of RAM per CPU core
%global _smp_tasksize_proc 4096

It doesn't look like a huge win from the packager's point of view, but I am
sure it's better to use the rpm upstream limits instead of external macros
such as constrain_build. 

-- 
Kalev
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-05 Thread Miroslav Suchý 

Dne 04. 10. 23 v 11:43 Martin Stransky napsal(a):

Hello guys,

Is there's a way how to set requested amount of ram for koji builders?

I'd like to use it as Firefox builds fail recently due low memory, like 
https://bugzilla.redhat.com/show_bug.cgi?id=2241690


Thanks,
Martin


Related - rpm has tunables that allows you to tune the build according the 
available resources. However memory is not there:

https://github.com/rpm-software-management/rpm/pull/2418

https://rpm-software-management.github.io/rpm/manual/buildprocess.html

--
Miroslav Suchy, RHCA
Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Martin Stransky 

On 10/5/23 08:03, Adam Williamson wrote:

On Wed, 2023-10-04 at 12:19 +0200, Kalev Lember wrote:

On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky  wrote:


Hello guys,

Is there's a way how to set requested amount of ram for koji builders?

I'd like to use it as Firefox builds fail recently due low memory, like
https://bugzilla.redhat.com/show_bug.cgi?id=2241690



I looked at the first linked build log in the ticket and it looks like the
build failed in the debuginfo extraction step. In that log, find-debuginfo
is called with -j3, and it could be that it ran out of memory because it
was trying to do 3 debuginfo extractions in parallel. You could try to
stick something like this in the spec file to tune it down:

# Require 32 GB of RAM per CPU core for debuginfo processing
%global _find_debuginfo_opts %limit_build -m 32768


Thanks for the idea, Kalev. I went ahead and applied this, and builds
for F38 and F39 succeeded. Let's hope it really helps and this wasn't
just luck!


Thanks Adam!

--
Martin Stransky
Software Engineer / Red Hat, Inc
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Adam Williamson 
On Wed, 2023-10-04 at 12:19 +0200, Kalev Lember wrote:
> On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky  wrote:
> 
> > Hello guys,
> > 
> > Is there's a way how to set requested amount of ram for koji builders?
> > 
> > I'd like to use it as Firefox builds fail recently due low memory, like
> > https://bugzilla.redhat.com/show_bug.cgi?id=2241690
> 
> 
> I looked at the first linked build log in the ticket and it looks like the
> build failed in the debuginfo extraction step. In that log, find-debuginfo
> is called with -j3, and it could be that it ran out of memory because it
> was trying to do 3 debuginfo extractions in parallel. You could try to
> stick something like this in the spec file to tune it down:
> 
> # Require 32 GB of RAM per CPU core for debuginfo processing
> %global _find_debuginfo_opts %limit_build -m 32768

Thanks for the idea, Kalev. I went ahead and applied this, and builds
for F38 and F39 succeeded. Let's hope it really helps and this wasn't
just luck!
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org
https://www.happyassassin.net



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Stephen Smoogen 
On Wed, 4 Oct 2023 at 09:48, Cristian Le via devel <
devel@lists.fedoraproject.org> wrote:

> Thank you Stephen,
>
> Didn't know we can have such control over koji scheduling. Is there any
> documentation on koji side or fedora packaging guidelines about this? E.g.
> what are the physical limits that can be queued, how to request specific
> number of CPUs for mpi packages etc.? Which macro sets take this into
> account? I believe %ctest ones do that, but I've found an MPI related
> situation where I need to override this.
>
> Uhm sorry, I am communicating poorly today it would seem. The packager
does not have any control on the number of CPUs, the amount of memory,
amount of disk space, etc. Those are set by infrastructure in how builders
are installed as virtual machines. Infrastructure in turn has to deal with
hard limits of how much disk space, cpu, memory are in the systems
infrastructure have which is further hard limited by the amount of rack
space, power, and budgets that they get handed to them. Most of these are
hard choices made by releng infrastructure on how to best divy up the
resources to deal with lots of small builds and regular large builds.

What a packager can control is how the package is configured and what
options are given to subparts of the 'build' inside of the RPM: config
options, make options, and debug size options.  A packager could choose to
use fewer CPUs by configuring their build to use 1 CPU (make -j1 ) or
possibly less memory, but it would need to be options set in whatever
equivalent to Makefiles etc that are there.

I hope that better explains things.. if it doesn't... I ask anyone else to
better explain things.

Also are the options equivalent in copr as well?
> On 2023/10/04 15:32, Stephen Smoogen wrote:
>
>
>
> On Wed, 4 Oct 2023 at 09:24, Cristian Le via devel <
> devel@lists.fedoraproject.org> wrote:
>
>> Hi Stephen,
>>
>> Could you also share the links for where the strategies for large builds
>> is, I am also curious about this? Is it within koji or another framework?
>>
>
> Sorry I meant the email Kalev Lember sent out
>
> # Require 32 GB of RAM per CPU core for debuginfo processing
> %global _find_debuginfo_opts %limit_build -m 32768
>
> ... and see if it helps. With this, it should pass -j1 to find-debuginfo
> when the builder has 32 GB RAM, -j2 when it has 64 GB etc.
>
> Other strategies are to limit the number of parallel parts of a build (aka
> make or cmake flags) happening at the same time
>
> Outside of that it is mainly up to a package and what it uses. The problem
> is that there are limited resources (number of builders, amount of memory
> per builder, amount of cpu) and as a packager you have to manage that
>
>
>> On 2023/10/04 14:41, Stephen Smoogen wrote:
>>
>>
>>
>> On Wed, 4 Oct 2023 at 05:44, Martin Stransky  wrote:
>>
>>> Hello guys,
>>>
>>> Is there's a way how to set requested amount of ram for koji builders?
>>>
>>>
>> No, the builders are mainly virtual machines on a set of hardware with a
>> set amount of memory per virtual machine. There are some 'real' hardware
>> servers but it is mainly a round robin to get them. As noted elsewhere
>> there are other strategies to use with large builds.
>>
>>
>
> --
> Stephen Smoogen, Red Hat Automotive
> Let us be kind to one another, for most of us are fighting a hard battle.
> -- Ian MacClaren
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
ht

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Cristian Le via devel 

Thank you Stephen,

Didn't know we can have such control over koji scheduling. Is there any 
documentation on koji side or fedora packaging guidelines about this? 
E.g. what are the physical limits that can be queued, how to request 
specific number of CPUs for mpi packages etc.? Which macro sets take 
this into account? I believe %ctest ones do that, but I've found an MPI 
related situation where I need to override this.


Also are the options equivalent in copr as well?

On 2023/10/04 15:32, Stephen Smoogen wrote:



On Wed, 4 Oct 2023 at 09:24, Cristian Le via devel 
 wrote:


Hi Stephen,

Could you also share the links for where the strategies for large
builds is, I am also curious about this? Is it within koji or
another framework?


Sorry I meant the email Kalev Lember sent out

# Require 32 GB of RAM per CPU core for debuginfo processing
%global _find_debuginfo_opts %limit_build -m 32768

... and see if it helps. With this, it should pass -j1 to 
find-debuginfo when the builder has 32 GB RAM, -j2 when it has 64 GB etc.


Other strategies are to limit the number of parallel parts of a build 
(aka make or cmake flags) happening at the same time


Outside of that it is mainly up to a package and what it uses. The 
problem is that there are limited resources (number of builders, 
amount of memory per builder, amount of cpu) and as a packager you 
have to manage that


On 2023/10/04 14:41, Stephen Smoogen wrote:



On Wed, 4 Oct 2023 at 05:44, Martin Stransky
 wrote:

Hello guys,

Is there's a way how to set requested amount of ram for koji
builders?


No, the builders are mainly virtual machines on a set of hardware
with a set amount of memory per virtual machine. There are some
'real' hardware servers but it is mainly a round robin to get
them. As noted elsewhere there are other strategies to use with
large builds.




--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard 
battle. -- Ian MacClaren


___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of 
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List 
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report 
it:https://pagure.io/fedora-infrastructure/new_issue___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Stephen Smoogen 
On Wed, 4 Oct 2023 at 09:24, Cristian Le via devel <
devel@lists.fedoraproject.org> wrote:

> Hi Stephen,
>
> Could you also share the links for where the strategies for large builds
> is, I am also curious about this? Is it within koji or another framework?
>

Sorry I meant the email Kalev Lember sent out

# Require 32 GB of RAM per CPU core for debuginfo processing
%global _find_debuginfo_opts %limit_build -m 32768

... and see if it helps. With this, it should pass -j1 to find-debuginfo
when the builder has 32 GB RAM, -j2 when it has 64 GB etc.

Other strategies are to limit the number of parallel parts of a build (aka
make or cmake flags) happening at the same time

Outside of that it is mainly up to a package and what it uses. The problem
is that there are limited resources (number of builders, amount of memory
per builder, amount of cpu) and as a packager you have to manage that


> On 2023/10/04 14:41, Stephen Smoogen wrote:
>
>
>
> On Wed, 4 Oct 2023 at 05:44, Martin Stransky  wrote:
>
>> Hello guys,
>>
>> Is there's a way how to set requested amount of ram for koji builders?
>>
>>
> No, the builders are mainly virtual machines on a set of hardware with a
> set amount of memory per virtual machine. There are some 'real' hardware
> servers but it is mainly a round robin to get them. As noted elsewhere
> there are other strategies to use with large builds.
>
>

-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Cristian Le via devel 

Hi Stephen,

Could you also share the links for where the strategies for large builds 
is, I am also curious about this? Is it within koji or another framework?


On 2023/10/04 14:41, Stephen Smoogen wrote:



On Wed, 4 Oct 2023 at 05:44, Martin Stransky  wrote:

Hello guys,

Is there's a way how to set requested amount of ram for koji builders?


No, the builders are mainly virtual machines on a set of hardware with 
a set amount of memory per virtual machine. There are some 'real' 
hardware servers but it is mainly a round robin to get them. As noted 
elsewhere there are other strategies to use with large builds.


I'd like to use it as Firefox builds fail recently due low memory,
like
https://bugzilla.redhat.com/show_bug.cgi?id=2241690

Thanks,
Martin

-- 
Martin Stransky

Software Engineer / Red Hat, Inc
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue



--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard 
battle. -- Ian MacClaren


___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of 
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List 
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report 
it:https://pagure.io/fedora-infrastructure/new_issue___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Stephen Smoogen 
On Wed, 4 Oct 2023 at 05:44, Martin Stransky  wrote:

> Hello guys,
>
> Is there's a way how to set requested amount of ram for koji builders?
>
>
No, the builders are mainly virtual machines on a set of hardware with a
set amount of memory per virtual machine. There are some 'real' hardware
servers but it is mainly a round robin to get them. As noted elsewhere
there are other strategies to use with large builds.



> I'd like to use it as Firefox builds fail recently due low memory, like
> https://bugzilla.redhat.com/show_bug.cgi?id=2241690
>
> Thanks,
> Martin
>
> --
> Martin Stransky
> Software Engineer / Red Hat, Inc
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Specify koji build machine mem req via. spec file

2023-10-04 Thread Kalev Lember 
On Wed, Oct 4, 2023 at 11:44 AM Martin Stransky  wrote:

> Hello guys,
>
> Is there's a way how to set requested amount of ram for koji builders?
>
> I'd like to use it as Firefox builds fail recently due low memory, like
> https://bugzilla.redhat.com/show_bug.cgi?id=2241690


I looked at the first linked build log in the ticket and it looks like the
build failed in the debuginfo extraction step. In that log, find-debuginfo
is called with -j3, and it could be that it ran out of memory because it
was trying to do 3 debuginfo extractions in parallel. You could try to
stick something like this in the spec file to tune it down:

# Require 32 GB of RAM per CPU core for debuginfo processing
%global _find_debuginfo_opts %limit_build -m 32768

... and see if it helps. With this, it should pass -j1 to find-debuginfo
when the builder has 32 GB RAM, -j2 when it has 64 GB etc.

If that doesn't work, then there's also the option of using -g1 to reduce
the binary sizes so they take less memory to process (but that makes
backtraces much worse so I'd only leave it for emergencies if you can't get
an important build out otherwise).

Another option is that there's a heavybuilder channel in koji, which has
beefier machines, but it has some significant capacity issues (only 2
aarch64 builders, iirc) so it would be best to avoid it, if possible.
Chromium and webkitgtk are using it and sometimes builds have to wait a
long time after each other, especially when building something for all
Fedora branches.

Hope this helps,
Kalev
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue