Hi Paul, On Sun, Oct 3, 2021 at 12:39 PM Paul Gevers <elb...@debian.org> wrote:
> Hi Reinhard, > > > The consistent OOM is surprising given that you state that the worker > > has 250GB of RAM. Looking at the logs, > > I note that the tests are being passed the option -p 160 by the > > dh-golang helper, so it will build > > and run test executables concurrently. That confirms to me that we are > > indeed running on these 250GB/160 core workers. > > We only have one armhf host, so *all* armhf tests are run there. I'm > also not seeing this problem back in the logs: > > https://ci.debian.net/munin/ci-worker-armel-01/ci-worker-armel-01/memory.html > . > But maybe the incident is too short to be caught by the once per 5 > minutes poll. > I think this might be the case. The whole package fails (or passes) usually in less than 5 minutes. These incidents would be very, very spiky. Probably something that captures the maximum values in the 5 minute interval would be more useful here. > Is it possible that armhf is setting up ulimits that limits the amount > > of memory the test may allocate? > > root@ci-worker-armel-01:~# ulimit > unlimited > > > As far as I understand, all golang packages use autodep8 to declare the > > tests, > > which doesn't support adding the Architecture field. > > I think you're right, but maybe we should fix autodep8 to enable the > change like the other overwrites in section `PACKAGE-SPECIFIC > CONFIGURATION` of $(man autodep8). > > > In order to get > > around this, > > I guess I could remove the Testsuite field from debian/control and add a > > debian/tests/control that looks similar to what autodep8 generates, but > adds > > the Architecture: !armhf restriction. > > Maybe until autodep8 is fixed? I fear that this may cause future trouble > if/when autodep8 support for golang gets updated. > Sure, I've cobbled up https://salsa.debian.org/ci-team/autodep8/-/merge_requests/25 for this. I've went ahead and implemented my limiting idea anyways: https://salsa.debian.org/go-team/packages/golang-github-klauspost-compress/-/commit/836de264a300e102a54055c1e3d629f82a1e8a1b Maybe Adrian is right and with too many goroutines going on at the same time, the test executable does exhaust the 3GB memory limitation on a 32bit architecture. debhelper has this convenient --max-parallel switch that I used in combination with setting GOMAXPROCS=8 to limit the number of goroutines in the test. the most recent test on armhf passes with no noticeable slowdown. Maybe we could go higher than 8, but finding out the optimal number does not seem like a good use of our time. Thanks for your help and input! -- regards, Reinhard
