Re: s390x KOJI builders issue

2022-03-04 Thread Colin Walters
On Thu, Mar 3, 2022, at 4:25 PM, Colin Walters wrote: > On Wed, Mar 2, 2022, at 7:04 PM, Kevin Fenzi wrote: > >> * OOm killer looks and says... oh hey, I need to kill something. This >> kojid process/slice is taking up all the memory. >> * kojid is killed. > > If we replaced Koji's backend with

Re: s390x KOJI builders issue

2022-03-03 Thread Kevin Fenzi
...snip... > > If this is just s390x builders, I'd prefer to see if we cannot rebalance > them to just pass your builds. So, looking at it, we have 20 buildvm's > on a host with 256gb mem. I could bump them all from 10 to 12 without > overcommiting. I don't know if 2gb would help enough tho? Is

Re: s390x KOJI builders issue

2022-03-03 Thread Colin Walters
On Wed, Mar 2, 2022, at 7:04 PM, Kevin Fenzi wrote: > * OOm killer looks and says... oh hey, I need to kill something. This > kojid process/slice is taking up all the memory. > * kojid is killed. If we replaced Koji's backend with Kubernetes (at least my employer's production way to run Linux

Re: s390x KOJI builders issue

2022-03-03 Thread Kevin Fenzi
On Thu, Mar 03, 2022 at 02:32:50AM +0100, Michal Schorm wrote: > In many cases, the build is killed during compilation itself. > I'd understand the situation, if it would consistently fail somewhere > during the testsuite on OOM errors, but it's weirder than that. > > Until now, I didn't have

Re: s390x KOJI builders issue

2022-03-03 Thread Florian Weimer
* Kevin Fenzi: > Perhaps there's some way to adjust the oom killer to kill the build > instead of kojid? There is /proc/PID/oom_score_adj. I don't know how to use it, sorry. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To

Re: s390x KOJI builders issue

2022-03-02 Thread Michal Schorm
In many cases, the build is killed during compilation itself. I'd understand the situation, if it would consistently fail somewhere during the testsuite on OOM errors, but it's weirder than that. Until now, I didn't have this issue. Why now? The tests are still important. Through the years I

Re: s390x KOJI builders issue

2022-03-02 Thread Kevin Fenzi
On Wed, Mar 02, 2022 at 03:54:32PM +0100, Florian Weimer wrote: > * Michael Catanzaro: > > > On Wed, Mar 2 2022 at 02:21:22 PM +0100, Dan Horák > > wrote: > >> those are weird, the build tasks have been restarted many times by the > >> builder daemon, after something crashed there (OOM?) ... > >

Re: s390x KOJI builders issue

2022-03-02 Thread Florian Weimer
* Michael Catanzaro: > On Wed, Mar 2 2022 at 02:21:22 PM +0100, Dan Horák > wrote: >> those are weird, the build tasks have been restarted many times by the >> builder daemon, after something crashed there (OOM?) ... > > This was happening to me on armv7hl a few weeks ago. Kevin Fenzi >

Re: s390x KOJI builders issue

2022-03-02 Thread Michael Catanzaro
On Wed, Mar 2 2022 at 02:21:22 PM +0100, Dan Horák wrote: those are weird, the build tasks have been restarted many times by the builder daemon, after something crashed there (OOM?) ... This was happening to me on armv7hl a few weeks ago. Kevin Fenzi investigated and discovered that the

Re: s390x KOJI builders issue

2022-03-02 Thread Dan Horák
On Wed, 2 Mar 2022 14:08:23 +0100 Michal Schorm wrote: > Hello, > for the last few days, I'm not able to finish my builds of the > 'mariadb' package on s390x architecture. > > Those builds freeze, e.g. several of these: > https://koji.fedoraproject.org/koji/taskinfo?taskID=83297826 >

s390x KOJI builders issue

2022-03-02 Thread Michal Schorm
Hello, for the last few days, I'm not able to finish my builds of the 'mariadb' package on s390x architecture. Those builds freeze, e.g. several of these: https://koji.fedoraproject.org/koji/taskinfo?taskID=83297826 https://koji.fedoraproject.org/koji/taskinfo?taskID=83296979