bug#40790: OOM error in graphical installer tests.

[Mathieu Othacehe]

Hello,

The "gui-installed-os" test fails on master because "guix system init"
command is OOM killed. Log of the failure is attached.

Thanks,

Mathieu


oom.log
Description: Binary data

bug#40790: OOM error in graphical installer tests.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe  skribis:

> The "gui-installed-os" test fails on master because "guix system init"
> command is OOM killed. Log of the failure is attached.

[...]

> Apr 23 11:43:27 localhost installer[230]: command ("guix" "system" "init" 
> "--fallback" "--no-grafts" "--no-substitutes" "/mnt/etc/config.scm" "/mnt") 
> failed with exit code #finstaller[230]: Umounted /remove suc 

Can we throw a little bit more memory at it?  :-)

Is there a commit that trigger this failure?

Thanks,
Ludo’.

bug#40790: OOM error in graphical installer tests.

[Mathieu Othacehe]

Hello,

> Can we throw a little bit more memory at it?  :-)

That's what I did with ae1a0f758 :)

> Is there a commit that trigger this failure?

I didn't have the courage to bisect it, given how long this test
is. Fixing it, I discovered two more issues:

* "sherpherd services" test was failing because a wrong swap device was
  used (fixed with be3abf28a).

* I have some hangs when doing the final testing in marionette. Still
  trying to understand why.

Thanks,

Mathieu

bug#40790: OOM error in graphical installer tests.

[Mathieu Othacehe]

> * I have some hangs when doing the final testing in marionette. Still
>   trying to understand why.

Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78. Still an issue
remaining. "gui-installed-desktop-os-encrypted" closure is about 5.5GiB
which makes too much files for libisofs:

--8<---cut here---start->8---
xorriso : UPDATE :  427934 files added in 19 seconds
libisofs: FAILURE : HFS+ map nodes aren't implemented
libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names
xorriso : FAILURE : Failed to prepare session write run
xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE
--8<---cut here---end--->8---

Not sure how to fix it. Someone?

Thanks,

Mathieu

bug#40790: OOM error in graphical installer tests.

[Ludovic Courtès]

Hi,

Mathieu Othacehe  skribis:

>> Can we throw a little bit more memory at it?  :-)
>
> That's what I did with ae1a0f758 :)

Oh, good.

Next we should of course find out why ‘guix system init’ is consuming so
much memory.  I haven’t tried GC-profiling that yet, but here’s a rough
figure:

--8<---cut here---start->8---
$ GUIX_PROFILING=gc GUIX_PROFILING_EVENTS=exit guix system build -d --no-grafts 
gnu/system/examples/desktop.tmpl
/gnu/store/2by26x4pvvp9vi1f0lmrq39wqqkj0s0f-system.drv
Garbage collection statistics:
  heap size:98.80 MiB
  allocated:548.35 MiB
  GC times: 26
  time spent in GC: 1.84 seconds (36% of user time)
$ guix describe
Generacio 140   Apr 29 2020 19:43:22(nuna)
  guix 19987f5
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 19987f5efe0acb65918802681f3a18cbb6a07c2a
--8<---cut here---end--->8---

So at first sight, it’s surprising that it couldn’t fit in the VM with
800 MiB of RAM.  Oh well, we’ll see!

Nitpick: please include the canonical bug URL in commit logs rather
than a link to the mailing list, like so:

  Fixes .

That makes it easier to grep for fixed bugs.

> I didn't have the courage to bisect it, given how long this test
> is. Fixing it, I discovered two more issues:
>
> * "sherpherd services" test was failing because a wrong swap device was
>   used (fixed with be3abf28a).

Good catch!

> * I have some hangs when doing the final testing in marionette. Still
>   trying to understand why.

Oh.

Thank you!

Ludo’.

bug#40790: OOM error in graphical installer tests.

[Bengt Richter]

Hi Ludo, et al,

On +2020-04-30 23:10:05 +0200, Ludovic Courtès wrote:
[...]

rc1: best-practice: commit logs: message syntax
--8<---cut here---start->8---
> Nitpick: please include the canonical bug URL in commit logs rather
> than a link to the mailing list, like so:
> 
>   Fixes .
> 
> That makes it easier to grep for fixed bugs.
--8<---cut here---end--->8---

ISTM your "nit-pick" is a good example of an embedded mail-list nugget.

Here's my wish:
If you and other developers notice (without false modesty ;-) that you
are posting a good example of something, then if you would enclose it
in guix snippet-delimiters as above, with a single prefixed line
starting with "primary-focus-subject:" and optionally followed by other
greppable (or list-matchable) sub-categories or ad-hoc words or names or tags,
then these nuggets would be easy to extract automatically, to
create a treasure-trove of searchable useful goodies.

I would hope to see useful one-liners and examples of package definition,
system configs, debugging methods, small scripts, and good info-URLs etc.
delimited and prefixed.

(The "primary-focus-subject:" of "rc1:" would permit us to invent improved
syntaxes and evolve them as "rc2" etc until we settle on a canonical set
for primary foci, (and PEG grammar for the rest, if we want to get fancy).

Perhaps the servers could have a gem-and-nugget-bot authoring an html
blog of these things, with automatic attibutions etc from email headers?

-- 
Regards,
Bengt Richter

bug#40790: OOM error in graphical installer tests.

[Ludovic Courtès]

Hi,

Mathieu Othacehe  skribis:

>> * I have some hangs when doing the final testing in marionette. Still
>>   trying to understand why.
>
> Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78.

Cool, sorry for breaking things.  :-/

> Still an issue remaining. "gui-installed-desktop-os-encrypted" closure
> is about 5.5GiB which makes too much files for libisofs:
>
> xorriso : UPDATE :  427934 files added in 19 seconds
> libisofs: FAILURE : HFS+ map nodes aren't implemented
> libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file 
> names
> xorriso : FAILURE : Failed to prepare session write run
> xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE
>
> Not sure how to fix it. Someone?

I tried the attached patch, which would select just one desktop
environments, thinking it might appease Xorriso, but actually it still
OOMs at the very last step:

--8<---cut here---start->8---
xorriso : UPDATE :  141879 file filters processed in 848 seconds
xorriso : UPDATE :  142026 file filters processed in 849 seconds
xorriso : UPDATE :  142102 file filters processed in 850 seconds
xorriso : NOTE : Copying to System Area: 512 bytes from file 
'/gnu/store/dvfl5ax7pj6c62f1ccc27qcrcb8nafr7-grub-hybrid-2.04/lib/grub/i386-pc/boot_hybrid.img'
[13266.747348] xorriso invoked oom-killer: 
gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13266.749415] CPU: 0 PID: 251 Comm: xorriso Not tainted 5.4.36-gnu #1
[13266.750777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
--8<---cut here---end--->8---

Testing again with this extra patch:

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 6f81ac16ff..9f240b50ee 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -356,7 +356,7 @@ INPUTS is a list of inputs (as for packages)."
#:substitutable? substitutable?
 
;; Xorriso seems to be quite memory-hungry, so increase the VM's RAM size.
-   #:memory-size 512))
+   #:memory-size 1024))
 
 (define* (qemu-image #:key
  (name "qemu-image")

I wonder if why it’s taking so much memory, and whether there’s
something else at play, like temporary files written to the overlaid
tmpfs and thus eating memory.

Thoughts?

Ludo’.

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 019e21fd39..941958d698 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -957,7 +957,7 @@ build (current-guix) and then store a couple of full system images.")
 
 (define* (gui-test-program marionette
#:key
-   (desktop? #f)
+   (desktops '())
(encrypted? #f))
   #~(let ()
   (define (screenshot file)
@@ -971,7 +971,9 @@ build (current-guix) and then store a couple of full system images.")
   (setvbuf (current-output-port) 'none)
   (setvbuf (current-error-port) 'none)
 
-  (marionette-eval* '(use-modules (gnu installer tests))
+  (marionette-eval* '(use-modules (gnu installer tests)
+  (srfi srfi-1)
+  (srfi srfi-26))
 #$marionette)
 
   ;; Arrange so that 'converse' prints debugging output to the console.
@@ -1016,7 +1018,9 @@ build (current-guix) and then store a couple of full system images.")
 
   (marionette-eval* '(choose-services installer-socket
   #:choose-desktop-environment?
-  (const #$desktop?)
+  (lambda (desktop)
+(any (cut string-contains desktop <>)
+ '#$desktops))
   #:choose-network-service?
   (const #f))
 #$marionette)
@@ -1110,7 +1114,7 @@ build (current-guix) and then store a couple of full system images.")
 
 (define* (guided-installation-test name
#:key
-   (desktop? #f)
+   (desktops '())
(encrypted? #f)
target-os
(install-size 'guess)
@@ -1132,7 +1136,7 @@ build (current-guix) and then store a couple of full system images.")
(lambda (marionette)
  (gui-test-program
   marionette
-  #:desktop? desktop?
+  #:desktops desktops
   #:encrypted? encrypted?
  (command (qemu-command/writable-image image)))
   (run-basic-test target-os command nam