Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> Go for it!

Done!  Thanks for the review.

> Thank you, and apologies for the looong delay!

No problem, and apologies on my side too for the delays.

> Ludo’.

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Returning to this issue after a long pause, I suddenly had a new idea,
> which is to use something akin to 'use-modules' instead of 'load' on the
> files.  (I need a procedure and not a macro, and don't actually need to
> 'use' the modules, so I use 'resolve-interface' in the patch below.
> Don't know if there's a better suited procedure for this purpose.)

Sounds good.

> The tricky part is turning file names into module names,

If that’s the only tricky part, I think we’re doing OK.  ;-)

> This entails the least amount of duplicated effort from all the variants
> we had so far, and finishes in below 2 minutes on my machine.  (I think
> the only remaining duplication is the compilation of the modules during
> 'resolve-interface' and their compilation via 'compile-file', and the
> former cannot be done in parallel, so we could again cut the time to
> below half of what it is now, if we solved that.)

Not bad!

I just tried it on my laptop, and it was indeed fast.

> From 77ac65593a94673872c477bbbd18eb2465c76030 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  
> Date: Fri, 27 Nov 2015 09:27:55 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process, report-build-progress)
> (p-for-each): Remove.
> (build-guix): Load and compile files in one process.

Go for it!

Thank you, and apologies for the looong delay!

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> After some tinkering around I realized that the problem is that our
> workaround of loading files explicitly causes the package record
> type to be redefined after some packages have already been defined.
> More generally, we force the top-level of many files to be
> re-executed after they've already been executed as a result of a
> module import...
>
> Returning to this issue after a long pause, I suddenly had a new idea,
> which is to use something akin to 'use-modules' instead of 'load' on the
> files.  (I need a procedure and not a macro, and don't actually need to
> 'use' the modules, so I use 'resolve-interface' in the patch below.
> Don't know if there's a better suited procedure for this purpose.)
>
> The tricky part is turning file names into module names, but so far all
> seems unproblematic, so a straightforward transformation from the string
> "$out/foo/bar.scm" to the list (foo bar) does the job.  (Possible issues
> would be when a file name doesn't match a module name, or a file doesn't
> contain a module at all.)
>
> This entails the least amount of duplicated effort from all the variants
> we had so far, and finishes in below 2 minutes on my machine.  (I think
> the only remaining duplication is the compilation of the modules during
> 'resolve-interface' and their compilation via 'compile-file', and the
> former cannot be done in parallel, so we could again cut the time to
> below half of what it is now, if we solved that.)
>
> Tell me if you see any problems with this variant.

Pinging this thread since it seems to have been forgotten. :-)

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

 After some tinkering around I realized that the problem is that our
 workaround of loading files explicitly causes the package record
 type to be redefined after some packages have already been defined.
 More generally, we force the top-level of many files to be
 re-executed after they've already been executed as a result of a
 module import...

Returning to this issue after a long pause, I suddenly had a new idea,
which is to use something akin to 'use-modules' instead of 'load' on the
files.  (I need a procedure and not a macro, and don't actually need to
'use' the modules, so I use 'resolve-interface' in the patch below.
Don't know if there's a better suited procedure for this purpose.)

The tricky part is turning file names into module names, but so far all
seems unproblematic, so a straightforward transformation from the string
"$out/foo/bar.scm" to the list (foo bar) does the job.  (Possible issues
would be when a file name doesn't match a module name, or a file doesn't
contain a module at all.)

This entails the least amount of duplicated effort from all the variants
we had so far, and finishes in below 2 minutes on my machine.  (I think
the only remaining duplication is the compilation of the modules during
'resolve-interface' and their compilation via 'compile-file', and the
former cannot be done in parallel, so we could again cut the time to
below half of what it is now, if we solved that.)

Tell me if you see any problems with this variant.

>From 77ac65593a94673872c477bbbd18eb2465c76030 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Fri, 27 Nov 2015 09:27:55 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process, report-build-progress)
(p-for-each): Remove.
(build-guix): Load and compile files in one process.
---
 guix/build/pull.scm | 149 +++-
 1 file changed, 55 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..4ddb12a 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -33,75 +35,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1
-(pid
- #t)))
-
-(define* (report-build-progress total completed cont
-#:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-  (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel.  Call PROGRESS at each step, passing it
-the continuation.  Raise an error if one of the processes exit with non-zero."
-  (define total
-(length lst))
-
-  (define (wait-for-one-process)
-(match (waitpid WAIT_ANY)
-  ((_ . status)
-   (unless (zero? (status:exit-val status))
- (error "process failed" proc status)
-
-  (let loop ((lst   lst)
- (running   0)
- (completed 0))
-(match lst
-  (()
-   (or (zero? running)
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
- (loop lst running completed))
-  ((head . tail)
-   (if (< running max-processes)
-   (let ((running (+ 1 running)))
- (call-with-process (cut proc head))
- (progress total completed
-   (lambda ()
- (loop tail running completed
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
-   

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

Mark H Weaver  skribis:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> It would be great if the whole circular import problem could somehow be
>> solved by Guile (no idea how feasible it is).
>
> I think we should eliminate circular module dependencies.  They cause
> nasty problems, and there's no compelling reason that we need them,
> since our package dependency graph is necessarily a DAG.

I think that this would mean having essentially one file per package,
which in turns would add a lot of I/O overhead.

More importantly, the problems we are facing here have nothing to do
with circular dependencies.  The main issue remains
.

The next issue would be multi-threaded compilation, which I think may be
fragile due to unsynchronized accesses to module objects and obarrays,
as discussed in this thread.

WDYT?

I would be happy to see more eyeballs looking into it!  :-)

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Mark H Weaver  writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> It would be great if the whole circular import problem could somehow be
>> solved by Guile (no idea how feasible it is).
>
> I think we should eliminate circular module dependencies.  They cause
> nasty problems, and there's no compelling reason that we need them,
> since our package dependency graph is necessarily a DAG.
>
> We can eliminate the circular dependencies by breaking up our package
> modules into smaller pieces.  One package per module would trivially
> accomplish this, although I guess that's going too far.
>
> We might want to think about what tools could help us discover a much
> smaller number of package splittings that would eliminate the cycles,
> and to ensure that they would never again be introduced.

There might also be non-package module cycles, or more likely, cycles
involving package and non-package modules.  I think I encountered the
latter while sniffing around, though I don't remember exactly.  These
might be less trivial to fix...  Just an FYI.  Otherwise +1.

I don't know when my productivity will go back up (health issues), but
when it does and nobody's done it yet I might work on such a tool for
package modules.  If anyone else wants to then don't hold back though.

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Mark H Weaver]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> It would be great if the whole circular import problem could somehow be
> solved by Guile (no idea how feasible it is).

I think we should eliminate circular module dependencies.  They cause
nasty problems, and there's no compelling reason that we need them,
since our package dependency graph is necessarily a DAG.

We can eliminate the circular dependencies by breaking up our package
modules into smaller pieces.  One package per module would trivially
accomplish this, although I guess that's going too far.

We might want to think about what tools could help us discover a much
smaller number of package splittings that would eliminate the cycles,
and to ensure that they would never again be introduced.

  Mark

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>>
>>> After some tinkering around I realized that the problem is that our
>>> workaround of loading files explicitly causes the package record type to
>>> be redefined after some packages have already been defined.  More
>>> generally, we force the top-level of many files to be re-executed after
>>> they've already been executed as a result of a module import...
>>>
>>> It would be great if the whole circular import problem could somehow be
>>> solved by Guile (no idea how feasible it is).  On the meanwhile, we'll
>>> have to work around problems introduced by workarounds. :-)
>>
>> I think we’ve gone way too far in kludgeland.  ;-)
>>
>> The other idea I had, and which is known to work, it to rely exclusively
>> on auto-compilation.  By definition, Guile’s auto-compilation mode will
>> compile things in the right order, thereby allowing us to compile
>> everything in one process.  Probably not parallelizable though, since
>> this is under Guile’s control, not ours.
>>
>> Would you like to play with that?  Otherwise I can give it a try later.
>
> I looked around in guix/scripts/pull.scm and build-aux/build-self.scm,
> but couldn't figure out how to enable auto-compilation here.  I expected
> there would be a place where guile is called with --no-auto-compile.
> What am I missing?

We could do (set! %fresh-auto-compile #t), and also set XDG_CACHE_HOME
so we control where .go files go.

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> After some tinkering around I realized that the problem is that our
>> workaround of loading files explicitly causes the package record type to
>> be redefined after some packages have already been defined.  More
>> generally, we force the top-level of many files to be re-executed after
>> they've already been executed as a result of a module import...
>>
>> It would be great if the whole circular import problem could somehow be
>> solved by Guile (no idea how feasible it is).  On the meanwhile, we'll
>> have to work around problems introduced by workarounds. :-)
>
> I think we’ve gone way too far in kludgeland.  ;-)
>
> The other idea I had, and which is known to work, it to rely exclusively
> on auto-compilation.  By definition, Guile’s auto-compilation mode will
> compile things in the right order, thereby allowing us to compile
> everything in one process.  Probably not parallelizable though, since
> this is under Guile’s control, not ours.
>
> Would you like to play with that?  Otherwise I can give it a try later.

I looked around in guix/scripts/pull.scm and build-aux/build-self.scm,
but couldn't figure out how to enable auto-compilation here.  I expected
there would be a place where guile is called with --no-auto-compile.
What am I missing?

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> After some tinkering around I realized that the problem is that our
> workaround of loading files explicitly causes the package record type to
> be redefined after some packages have already been defined.  More
> generally, we force the top-level of many files to be re-executed after
> they've already been executed as a result of a module import...
>
> It would be great if the whole circular import problem could somehow be
> solved by Guile (no idea how feasible it is).  On the meanwhile, we'll
> have to work around problems introduced by workarounds. :-)

I think we’ve gone way too far in kludgeland.  ;-)

The other idea I had, and which is known to work, it to rely exclusively
on auto-compilation.  By definition, Guile’s auto-compilation mode will
compile things in the right order, thereby allowing us to compile
everything in one process.  Probably not parallelizable though, since
this is under Guile’s control, not ours.

Would you like to play with that?  Otherwise I can give it a try later.

Thank you,
Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> l...@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>>?: 2 [primitive-load 
>>> "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest/guix/scripts/import/hackage.scm"]
>>> In ice-9/eval.scm:
>>>  453: 1 Exception thrown while printing backtrace:
>>> ERROR: In procedure package-location: Wrong type argument: Error while 
>>> printing exception.
>>>
>>> ice-9/eval.scm:387:11: In procedure eval:
>>> ice-9/eval.scm:387:11: In procedure package-version: Wrong type argument: 
>>> Error while printing exception.
>>> builder for `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' 
>>> failed with exit code 1
>>> guix pull: error: build failed: build of 
>>> `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' failed
>>>
>>> Any idea?
>>>
>>> To me it sounds like there are two  record type descriptors in
>>> the wild, which is why ‘package-location’ in the package record printer
>>> bails out.
>>
>> That's one of the errors that result from a "bad" order of compiling the
>> files and when the 'load' hack isn't used to work around it, which isn't
>> the case in that patch...  Indeed I can't seem to reproduce the issue.
>>
>> The attached patch below also builds on the quoted patch, removes the
>> thread-safe-port procedure, and just sets the warning port to a void
>> port.  Applied on top of the current master, it works for me.
>
> On top of current master, it fails for me in the same way as above.
>
> To be clear, I applied the patch, ran ‘make dist’, and then:
>
>   time ./pre-inst-env guix pull --url=file://$PWD/guix-0.9.0.tar.gz
>
> Are you doing the same?  The “loading” part is done sequentially, so it
> should be deterministic.

Whoops, I had not rerun the whole 'make dist' after rebasing on master,
only copied the new guix/build/pull.scm into an existing tarball (I had
gotten used to doing that because it saves time while working on a
single file), so changes in other files were missing.

After some tinkering around I realized that the problem is that our
workaround of loading files explicitly causes the package record type to
be redefined after some packages have already been defined.  More
generally, we force the top-level of many files to be re-executed after
they've already been executed as a result of a module import...

It would be great if the whole circular import problem could somehow be
solved by Guile (no idea how feasible it is).  On the meanwhile, we'll
have to work around problems introduced by workarounds. :-) Moving the
loading of guix/package.scm to the very front seems to solve the issue.
Other record types could still cause the same issue, but their relative
rarity of use hopefully makes this a non-issue.  I also moved the
loading of guix/ files before gnu/ files again, which might also help
with that.  (For package.scm it wasn't sufficient, probably because some
modules under guix/ import some gnu package modules, before package.scm
is loaded explicitly.)

One can imagine a wholly more robust version of the workaround, which
avoids the re-execution of top-levels.  A variant of load[-primitive]
that doesn't load a file again if it was already loaded would do.
That's basically what importing a module does, so scanning for module
definitions in files and importing them might work, but seems somewhat
hacky...  For now, here's the patch that just loads package.scm first.

>From dcb563f611c4fbd6e3e22106c60626f32c04f9e5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Fri, 27 Nov 2015 09:27:55 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process, report-build-progress)
(p-for-each): Remove.
(build-guix): Load and compile files in one process.
---
 guix/build/pull.scm | 149 +++-
 1 file changed, 55 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..3025442 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -33,75 +35,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primi

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> l...@gnu.org (Ludovic Courtès) writes:

[...]

>>?: 2 [primitive-load 
>> "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest/guix/scripts/import/hackage.scm"]
>> In ice-9/eval.scm:
>>  453: 1 Exception thrown while printing backtrace:
>> ERROR: In procedure package-location: Wrong type argument: Error while 
>> printing exception.
>>
>> ice-9/eval.scm:387:11: In procedure eval:
>> ice-9/eval.scm:387:11: In procedure package-version: Wrong type argument: 
>> Error while printing exception.
>> builder for `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' 
>> failed with exit code 1
>> guix pull: error: build failed: build of 
>> `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' failed
>>
>> Any idea?
>>
>> To me it sounds like there are two  record type descriptors in
>> the wild, which is why ‘package-location’ in the package record printer
>> bails out.
>
> That's one of the errors that result from a "bad" order of compiling the
> files and when the 'load' hack isn't used to work around it, which isn't
> the case in that patch...  Indeed I can't seem to reproduce the issue.
>
> The attached patch below also builds on the quoted patch, removes the
> thread-safe-port procedure, and just sets the warning port to a void
> port.  Applied on top of the current master, it works for me.

On top of current master, it fails for me in the same way as above.

To be clear, I applied the patch, ran ‘make dist’, and then:

  time ./pre-inst-env guix pull --url=file://$PWD/guix-0.9.0.tar.gz

Are you doing the same?  The “loading” part is done sequentially, so it
should be deterministic.

> Thanks for picking this up and sorry that I couldn't finish it. :-)

No problem, we’re getting there!  :-)

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> From 78be6d09d2d4c0a563be14c66ac2a1a345ff9b1d Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>>  
>> Date: Thu, 5 Nov 2015 23:43:20 +0100
>> Subject: [PATCH] build: pull: Compile .scm files in one process.
>>
>> * guix/build/pull.scm (call-with-process, report-build-progress)
>> (p-for-each): Remove.
>> (thread-safe-port): New procedure.
>> (build-guix): Load and compile files in one process.
>
> Just tried this patch without the ‘thread-safe-port’ procedure, but I
> got this (current master):
>
> loading... 95.4% of 474 filesBacktrace:
> In ice-9/boot-9.scm:
>  157: 14 [catch #t # ...]
> In unknown file:
>?: 13 [apply-smob/1 #]
> In ice-9/boot-9.scm:
>   63: 12 [call-with-prompt prompt0 ...]
> In ice-9/eval.scm:
>  432: 11 [eval # #]
> In ice-9/boot-9.scm:
> 2401: 10 [save-module-excursion # ()>]
> 4050: 9 [#]
> 1724: 8 [%start-stack load-stack # ice-9/boot-9.scm:4041:10 ()>]
> 1729: 7 [#]
> In unknown file:
>?: 6 [primitive-load 
> "/gnu/store/hx0jk73cx50f3vpi0yyrbn0pd8ws8m0v-guix-latest-builder"]
> In ./guix/build/pull.scm:
>   47: 5 [build-guix "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest" 
> ...]
>   91: 4 [#]
> In ice-9/boot-9.scm:
> 2401: 3 [save-module-excursion # ./guix/build/pull.scm:92:14 ()>]
> In unknown file:
>?: 2 [primitive-load 
> "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest/guix/scripts/import/hackage.scm"]
> In ice-9/eval.scm:
>  453: 1 Exception thrown while printing backtrace:
> ERROR: In procedure package-location: Wrong type argument: Error while 
> printing exception.
>
> ice-9/eval.scm:387:11: In procedure eval:
> ice-9/eval.scm:387:11: In procedure package-version: Wrong type argument: 
> Error while printing exception.
> builder for `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' 
> failed with exit code 1
> guix pull: error: build failed: build of 
> `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' failed
>
> Any idea?
>
> To me it sounds like there are two  record type descriptors in
> the wild, which is why ‘package-location’ in the package record printer
> bails out.

That's one of the errors that result from a "bad" order of compiling the
files and when the 'load' hack isn't used to work around it, which isn't
the case in that patch...  Indeed I can't seem to reproduce the issue.

The attached patch below also builds on the quoted patch, removes the
thread-safe-port procedure, and just sets the warning port to a void
port.  Applied on top of the current master, it works for me.

Maybe you applied a different patch by accident?

Thanks for picking this up and sorry that I couldn't finish it. :-)

>From 4cb8ad8006ba359e984f4b6e765be082b7d5f9c0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Fri, 27 Nov 2015 09:27:55 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process, report-build-progress)
(p-for-each): Remove.
(build-guix): Load and compile files in one process.
---
 guix/build/pull.scm | 142 ++--
 1 file changed, 48 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..e77a582 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -33,75 +35,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1
-(pid
- #t)))
-
-(define* (report-build-progress total completed cont
-#:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-  (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCE

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> From 78be6d09d2d4c0a563be14c66ac2a1a345ff9b1d Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  
> Date: Thu, 5 Nov 2015 23:43:20 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process, report-build-progress)
> (p-for-each): Remove.
> (thread-safe-port): New procedure.
> (build-guix): Load and compile files in one process.

Just tried this patch without the ‘thread-safe-port’ procedure, but I
got this (current master):

--8<---cut here---start->8---
loading...   95.4% of 474 filesBacktrace:
In ice-9/boot-9.scm:
 157: 14 [catch #t # ...]
In unknown file:
   ?: 13 [apply-smob/1 #]
In ice-9/boot-9.scm:
  63: 12 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 11 [eval # #]
In ice-9/boot-9.scm:
2401: 10 [save-module-excursion #]
4050: 9 [#]
1724: 8 [%start-stack load-stack #]
1729: 7 [#]
In unknown file:
   ?: 6 [primitive-load 
"/gnu/store/hx0jk73cx50f3vpi0yyrbn0pd8ws8m0v-guix-latest-builder"]
In ./guix/build/pull.scm:
  47: 5 [build-guix "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest" 
...]
  91: 4 [#]
In ice-9/boot-9.scm:
2401: 3 [save-module-excursion #]
In unknown file:
   ?: 2 [primitive-load 
"/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-latest/guix/scripts/import/hackage.scm"]
In ice-9/eval.scm:
 453: 1 Exception thrown while printing backtrace:
ERROR: In procedure package-location: Wrong type argument: Error while printing 
exception.

ice-9/eval.scm:387:11: In procedure eval:
ice-9/eval.scm:387:11: In procedure package-version: Wrong type argument: Error 
while printing exception.
builder for `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' 
failed with exit code 1
guix pull: error: build failed: build of 
`/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv' failed
--8<---cut here---end--->8---

Any idea?

To me it sounds like there are two  record type descriptors in
the wild, which is why ‘package-location’ in the package record printer
bails out.

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> l...@gnu.org (Ludovic Courtès) writes:

[...]

>> I think we’re in trouble if DEBUG-PORT is a real port because ports
>> aren’t thread-safe in 2.0, and ‘compile-file’ might write to it.  Void
>> ports seem to be OK because their ‘write’ method doesn’t do anything.
>>
>> So I think we have to do things sequentially when DEBUG-PORT is
>> non-void.
>>
>> I’m also concerned about modules accessed by concurrent calls to
>> ‘compile-file’ because modules and hash tables aren’t thread-safe
>> either.
>>
>> WDYT?
>>
>> Thanks to the two of you!  Sounds like we’ll soon have a less slow or
>> even a fast ‘guix pull’.  ;-)
>
> Hmm, how about this one?  I guess it could still garble output if
> 'compile-file' writes things in small pieces that belong together, but
> at least it's sure not to crash.  Compile-file doesn't seem to write a
> lot either; one probably won't see much garbled output in practice.
> Maybe the speed gain makes this acceptable.

It’s not about garbled output; it’s about concurrent modifications of
the port’s internal structure, which can lead to segfaults and whatnot.

> Re. modules, AFAIUI compile-file shouldn't do any further mutations on
> modules, since we load them in advance.  Maybe Andy can tell how safe
> this is.

Right, it *seems* reasonable.  Andy, Mark?

> From 78be6d09d2d4c0a563be14c66ac2a1a345ff9b1d Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  
> Date: Thu, 5 Nov 2015 23:43:20 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process, report-build-progress)
> (p-for-each): Remove.
> (thread-safe-port): New procedure.
> (build-guix): Load and compile files in one process.

[...]

> +(define (thread-safe-port port)
> +  (define mutex (make-mutex))
> +  (define (wrap proc)
> +(lambda args
> +  (with-mutex mutex (apply proc args
> +  (make-soft-port
> +   (vector (wrap write-char) (wrap display) (wrap force-output) #f #f) "w"))

This doesn’t address the fact that internally, things may still go wrong
(see ‘scm_lfwrite’ and the likes.)

I think we simply need to build sequentially when ports are involved,
which is when --debug is used.

Could you address this last bit?

Thanks!

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> From 0eaf7d4c264dc531718a4c0b933323f48ea91930 Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>>  
>> Date: Thu, 5 Nov 2015 23:43:20 +0100
>> Subject: [PATCH] build: pull: Compile .scm files in one process.
>>
>> * guix/build/pull.scm (call-with-process): Removed procedure.
>> (report-build-progress): Removed procedure.
>> (p-for-each): Removed procedure.
>
> Rather:
>
> … (call-with-process, report-build-progress, p-for-each): Remove.

Done.

>> +(let ((mutex (make-mutex))
>> +  (completed 0))
>> +  (par-for-each
>> +   (lambda (file)
>> + (with-mutex mutex
>> +   (display #\cr log-port)
>> +   (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: 
>> i18n
>> +   (* 100. (/ completed total)) total)
>> +   (force-output log-port)
>> +   (format debug-port "~%compiling '~a'...~%" file))
>> + (let ((go (string-append (string-drop-right file 4) ".go")))
>> +   (compile-file file
>> + #:output-file go
>> + #:opts %auto-compilation-options))
>> + (with-mutex mutex
>> +   (set! completed (+ 1 completed
>> +   files)
>
> Does it actually work reliably?  :-)
>
> I think we’re in trouble if DEBUG-PORT is a real port because ports
> aren’t thread-safe in 2.0, and ‘compile-file’ might write to it.  Void
> ports seem to be OK because their ‘write’ method doesn’t do anything.
>
> So I think we have to do things sequentially when DEBUG-PORT is
> non-void.
>
> I’m also concerned about modules accessed by concurrent calls to
> ‘compile-file’ because modules and hash tables aren’t thread-safe
> either.
>
> WDYT?
>
> Thanks to the two of you!  Sounds like we’ll soon have a less slow or
> even a fast ‘guix pull’.  ;-)

Hmm, how about this one?  I guess it could still garble output if
'compile-file' writes things in small pieces that belong together, but
at least it's sure not to crash.  Compile-file doesn't seem to write a
lot either; one probably won't see much garbled output in practice.
Maybe the speed gain makes this acceptable.

Re. modules, AFAIUI compile-file shouldn't do any further mutations on
modules, since we load them in advance.  Maybe Andy can tell how safe
this is.

>From 78be6d09d2d4c0a563be14c66ac2a1a345ff9b1d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Thu, 5 Nov 2015 23:43:20 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process, report-build-progress)
(p-for-each): Remove.
(thread-safe-port): New procedure.
(build-guix): Load and compile files in one process.
---
 guix/build/pull.scm | 147 +++-
 1 file changed, 54 insertions(+), 93 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..75bce7c 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -33,75 +35,18 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1
-(pid
- #t)))
-
-(define* (report-build-progress total completed cont
-#:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-  (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel.  Call PROGRESS at each step, passing it
-the continuation.  Raise an error if one of the processes exit with non-zero."
-  (define total
-(length lst))
-
-  (define (wait-for-one-process)
-(match (waitpid WAIT_ANY)
-  ((_ . s

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> With that and your further help on IRC, I ended up with the following
> patch.
>
> This takes about 2 minutes on my system.  The CPU gets fried during the
> parallel compilation, as expected. ;-)

Nice.  :-)

> From 0eaf7d4c264dc531718a4c0b933323f48ea91930 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  
> Date: Thu, 5 Nov 2015 23:43:20 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process): Removed procedure.
> (report-build-progress): Removed procedure.
> (p-for-each): Removed procedure.

Rather:

… (call-with-process, report-build-progress, p-for-each): Remove.

> +(let ((mutex (make-mutex))
> +  (completed 0))
> +  (par-for-each
> +   (lambda (file)
> + (with-mutex mutex
> +   (display #\cr log-port)
> +   (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: 
> i18n
> +   (* 100. (/ completed total)) total)
> +   (force-output log-port)
> +   (format debug-port "~%compiling '~a'...~%" file))
> + (let ((go (string-append (string-drop-right file 4) ".go")))
> +   (compile-file file
> + #:output-file go
> + #:opts %auto-compilation-options))
> + (with-mutex mutex
> +   (set! completed (+ 1 completed
> +   files)

Does it actually work reliably?  :-)

I think we’re in trouble if DEBUG-PORT is a real port because ports
aren’t thread-safe in 2.0, and ‘compile-file’ might write to it.  Void
ports seem to be OK because their ‘write’ method doesn’t do anything.

So I think we have to do things sequentially when DEBUG-PORT is
non-void.

I’m also concerned about modules accessed by concurrent calls to
‘compile-file’ because modules and hash tables aren’t thread-safe
either.

WDYT?

Thanks to the two of you!  Sounds like we’ll soon have a less slow or
even a fast ‘guix pull’.  ;-)

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo  writes:

> On Thu 12 Nov 2015 12:27, taylanbayi...@gmail.com (Taylan Ulrich 
> "Bayırlı/Kammer") writes:
>
>> First attempt: when I simply put
>>
>>   (for-each (lambda (file)
>>   (display file)
>>   (newline)
>>   (primitive-load file))
>> files)
>>
>> before the named let loop, I get the following output (interesting bit
>> at the end, but all output pasted just in case):
>
> You'll want to do the primitive-load in a
>
>   (save-module-excursion
> (lambda () (primitive-load file)))
>
> Andy

Heh, that was easy.

With that and your further help on IRC, I ended up with the following
patch.

This takes about 2 minutes on my system.  The CPU gets fried during the
parallel compilation, as expected. ;-)

>From 0eaf7d4c264dc531718a4c0b933323f48ea91930 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Thu, 5 Nov 2015 23:43:20 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process): Removed procedure.
(report-build-progress): Removed procedure.
(p-for-each): Removed procedure.
(build-guix): Don't create subprocesses to compile the .scm files.
---
 guix/build/pull.scm | 140 +---
 1 file changed, 46 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..e525b7d 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -22,6 +23,7 @@
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
+  #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
@@ -33,75 +35,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1
-(pid
- #t)))
-
-(define* (report-build-progress total completed cont
-#:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-  (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel.  Call PROGRESS at each step, passing it
-the continuation.  Raise an error if one of the processes exit with non-zero."
-  (define total
-(length lst))
-
-  (define (wait-for-one-process)
-(match (waitpid WAIT_ANY)
-  ((_ . status)
-   (unless (zero? (status:exit-val status))
- (error "process failed" proc status)
-
-  (let loop ((lst   lst)
- (running   0)
- (completed 0))
-(match lst
-  (()
-   (or (zero? running)
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
- (loop lst running completed))
-  ((head . tail)
-   (if (< running max-processes)
-   (let ((running (+ 1 running)))
- (call-with-process (cut proc head))
- (progress total completed
-   (lambda ()
- (loop tail running completed
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
- (loop lst running completed)
-
 (define* (build-guix out source
  #:key gcrypt
- (debug-port (%make-void-port "w")))
+ (debug-port (%make-void-port "w"))
+ (log-port (current-error-port)))
   "Build and install Guix in directory OUT using SOURCE, a directory
 containing the source code.  Write any debugging output to DEBUG-PORT."
   (setvbuf (current-output-port) _IOLBF)
@@ -130,33 +67,48 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
 (set! %load-path (cons out %load-path))
 (set! %load-compiled-path (cons out

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Thu 12 Nov 2015 12:27, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

> First attempt: when I simply put
>
>   (for-each (lambda (file)
>   (display file)
>   (newline)
>   (primitive-load file))
> files)
>
> before the named let loop, I get the following output (interesting bit
> at the end, but all output pasted just in case):

You'll want to do the primitive-load in a

  (save-module-excursion
(lambda () (primitive-load file)))

Andy

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo  writes:

> On Thu 12 Nov 2015 10:23, taylanbayi...@gmail.com (Taylan Ulrich 
> "Bayırlı/Kammer") writes:
>
>> +;; Compile the .scm files.  Also load every compiled file after writing 
>> it
>> +;; to work around  (FIXME).
>
> Hi :)  Quick question.  Why not primitive-load each file in order,
> loading them with the evaluator, and then compile them all in parallel
> threads?

Mainly just because I didn't want to muck much more with it since I'm
not very clear on what really goes on in Guile. :-)  If you can assist
me a bit, I'll gladly give it a go.

First attempt: when I simply put

  (for-each (lambda (file)
  (display file)
  (newline)
  (primitive-load file))
files)

before the named let loop, I get the following output (interesting bit
at the end, but all output pasted just in case):

/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/artwork.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/activation.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/file-systems.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/install.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/linux-boot.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/linux-container.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/linux-initrd.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/linux-modules.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/build/vm.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages.scm
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/abduco.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/abiword.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/acct.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/acl.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/admin.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/adns.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/aidc.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/algebra.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/apl.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/apr.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/asciidoc.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/aspell.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/attr.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/audacity.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/audio.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/autogen.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/autotools.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/avahi.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/avr.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/backup.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/base.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bash.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bdw-gc.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bioinformatics.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bison.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bittorrent.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/boost.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/bootstrap.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/busybox.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/calcurse.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/calendar.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/ccache.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/cdrom.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/certs.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/check.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/ci.scm
/gnu/store/4swmskqkhdblr62rj126hizfc3np45bw-guix-latest/gnu/packages/cmake.scm
/gnu/store/4swmskqkhdblr62rj126hiz

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Thu 12 Nov 2015 10:23, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

> +;; Compile the .scm files.  Also load every compiled file after writing 
> it
> +;; to work around  (FIXME).

Hi :)  Quick question.  Why not primitive-load each file in order,
loading them with the evaluator, and then compile them all in parallel
threads?

Andy

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> l...@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>> (gnu scripts environment) imports (gnu packages bash), so we end up
>>> loading a big bunch of (gnu packages …) modules.
>>>
>>> The problem is that (gnu packages commencement) refers to (gnu packages
>>> bash) from its top-level.  But here, we are loading (gnu packages bash)
>>> first, which somehow entails loading (gnu packages cross-base), which
>>> loads (gnu packages commencement), which sees a (gnu packages bash)
>>> module containing zero bindings.
>>>
>>> What’s unclear to me though is why (gnu packages cross-base) gets loaded
>>> in the first place.
>>>
>>> Could you investigate in that direction?
>>
>> It seems to be
>>
>> (guix scripts environment) -> (gnu system linux-container) ->
>> (gnu system) -> (gnu packages firmware) -> (gnu packages cross-base)
>
> Good catch.
>
> I’m not sure how to address that.  We could introduce lazy references
> with ‘module-ref’ at any of these stages.  That would work but sounds
> like a hack.  Hmm.  Ideas?  :-)

After our exchange on IRC, I reached this variant which only moves
gnu.scm to the end which avoids the progress report stall and the
circular import issue.

I think it's worrying that a mere change in order of compilation can
cause errors, but the speed gains of this hopefully make it an
acceptable temporary solution.

Thanks for the help!

>From ef1a1c6cb3f2abd7faa147468d2f062f10adf2b2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 
Date: Thu, 5 Nov 2015 23:43:20 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process): Removed procedure.
(report-build-progress): Removed procedure.
(p-for-each): Removed procedure.
(build-guix): Don't create subprocesses to compile the .scm files.
---
 guix/build/pull.scm | 127 ++--
 1 file changed, 33 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..da43a5a 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014 Ludovic Courtès 
+;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,75 +34,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-(0
- (catch #t
-   (lambda ()
- (thunk)
- (primitive-exit 0))
-   (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1
-(pid
- #t)))
-
-(define* (report-build-progress total completed cont
-#:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-  (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel.  Call PROGRESS at each step, passing it
-the continuation.  Raise an error if one of the processes exit with non-zero."
-  (define total
-(length lst))
-
-  (define (wait-for-one-process)
-(match (waitpid WAIT_ANY)
-  ((_ . status)
-   (unless (zero? (status:exit-val status))
- (error "process failed" proc status)
-
-  (let loop ((lst   lst)
- (running   0)
- (completed 0))
-(match lst
-  (()
-   (or (zero? running)
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
- (loop lst running completed))
-  ((head . tail)
-   (if (< running max-processes)
-   (let ((running (+ 1 running)))
- (call-with-process (cut proc head))
- (progress total completed
-   (lambda ()
- (loop tail running completed
-   (let ((running   (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
-   (lambda ()
- (loop lst running completed)
-
 (define* (build-guix out source
  #:key gcrypt
- (debug-port (%make-void-port "w")))
+ 

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> l...@gnu.org (Ludovic Courtès) writes:

[...]

>> (gnu scripts environment) imports (gnu packages bash), so we end up
>> loading a big bunch of (gnu packages …) modules.
>>
>> The problem is that (gnu packages commencement) refers to (gnu packages
>> bash) from its top-level.  But here, we are loading (gnu packages bash)
>> first, which somehow entails loading (gnu packages cross-base), which
>> loads (gnu packages commencement), which sees a (gnu packages bash)
>> module containing zero bindings.
>>
>> What’s unclear to me though is why (gnu packages cross-base) gets loaded
>> in the first place.
>>
>> Could you investigate in that direction?
>
> It seems to be
>
> (guix scripts environment) -> (gnu system linux-container) ->
> (gnu system) -> (gnu packages firmware) -> (gnu packages cross-base)

Good catch.

I’m not sure how to address that.  We could introduce lazy references
with ‘module-ref’ at any of these stages.  That would work but sounds
like a hack.  Hmm.  Ideas?  :-)

> It would be amazing if we could easily create graphs of Guile modules...

We can do it with ‘guild use2dot’ (with the caveat that we need to
explicitly load (guix gexp) before because otherwise it borks when it
sees #~ and the likes.)  However the resulting graph is pretty much
intractable.

That said, it may be nice to have a similar tool that we could also use
to build Makefile rules or to make a topological sort so ‘guix pull’
builds files in topological order.  But that’s more of an optimization,
I think.

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> compiling 
>> '/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/scripts/environment.scm'...
>> Backtrace:
>> In ice-9/boot-9.scm:
>> 2401: 19 [save-module-excursion #> ice-9/boot-9.scm:3066:17 ()>]
>> 3085: 18 [#]
>> In unknown file:
>>?: 17 [primitive-load-path "gnu/packages/cross-base" ...]
>> In ice-9/eval.scm:
>>  505: 16 [# (define-module # 
>> # ...)]
>> In ice-9/psyntax.scm:
>> 1106: 15 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
>>  989: 14 [scan ((define-module (gnu packages cross-base) #:use-module ...)) 
>> () ...]
>>  279: 13 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
>> In ice-9/eval.scm:
>>  411: 12 [eval # ()]
>> In ice-9/boot-9.scm:
>> 2951: 11 [define-module* (gnu packages cross-base) #:filename ...]
>> 2926: 10 [resolve-imports ((#) (#) (#) (#) ...)]
>> 2864: 9 [resolve-interface (gnu packages commencement) #:select ...]
>> 2789: 8 [#> autoload version #:key ensure)> # ...]
>> 3065: 7 [try-module-autoload (gnu packages commencement) #f]
>> 2401: 6 [save-module-excursion #> ice-9/boot-9.scm:3066:17 ()>]
>> 3085: 5 [#]
>> In unknown file:
>>?: 4 [primitive-load-path "gnu/packages/commencement" ...]
>> In ice-9/eval.scm:
>>  432: 3 Exception thrown while printing backtrace:
>> ERROR: In procedure package-location: Wrong type argument: Error while 
>> printing exception.
>
> (gnu scripts environment) imports (gnu packages bash), so we end up
> loading a big bunch of (gnu packages …) modules.
>
> The problem is that (gnu packages commencement) refers to (gnu packages
> bash) from its top-level.  But here, we are loading (gnu packages bash)
> first, which somehow entails loading (gnu packages cross-base), which
> loads (gnu packages commencement), which sees a (gnu packages bash)
> module containing zero bindings.
>
> What’s unclear to me though is why (gnu packages cross-base) gets loaded
> in the first place.
>
> Could you investigate in that direction?

It seems to be

(guix scripts environment) -> (gnu system linux-container) ->
(gnu system) -> (gnu packages firmware) -> (gnu packages cross-base)

It would be amazing if we could easily create graphs of Guile modules...

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> compiling 
> '/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/scripts/environment.scm'...
> Backtrace:
> In ice-9/boot-9.scm:
> 2401: 19 [save-module-excursion # ice-9/boot-9.scm:3066:17 ()>]
> 3085: 18 [#]
> In unknown file:
>?: 17 [primitive-load-path "gnu/packages/cross-base" ...]
> In ice-9/eval.scm:
>  505: 16 [# (define-module # 
> # ...)]
> In ice-9/psyntax.scm:
> 1106: 15 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
>  989: 14 [scan ((define-module (gnu packages cross-base) #:use-module ...)) 
> () ...]
>  279: 13 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
> In ice-9/eval.scm:
>  411: 12 [eval # ()]
> In ice-9/boot-9.scm:
> 2951: 11 [define-module* (gnu packages cross-base) #:filename ...]
> 2926: 10 [resolve-imports ((#) (#) (#) (#) ...)]
> 2864: 9 [resolve-interface (gnu packages commencement) #:select ...]
> 2789: 8 [# autoload version #:key ensure)> # ...]
> 3065: 7 [try-module-autoload (gnu packages commencement) #f]
> 2401: 6 [save-module-excursion # ice-9/boot-9.scm:3066:17 ()>]
> 3085: 5 [#]
> In unknown file:
>?: 4 [primitive-load-path "gnu/packages/commencement" ...]
> In ice-9/eval.scm:
>  432: 3 Exception thrown while printing backtrace:
> ERROR: In procedure package-location: Wrong type argument: Error while 
> printing exception.

(gnu scripts environment) imports (gnu packages bash), so we end up
loading a big bunch of (gnu packages …) modules.

The problem is that (gnu packages commencement) refers to (gnu packages
bash) from its top-level.  But here, we are loading (gnu packages bash)
first, which somehow entails loading (gnu packages cross-base), which
loads (gnu packages commencement), which sees a (gnu packages bash)
module containing zero bindings.

What’s unclear to me though is why (gnu packages cross-base) gets loaded
in the first place.

Could you investigate in that direction?

Thanks in advance,
Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Mon 09 Nov 2015 09:50, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

> Andy Wingo  writes:
>
>> On Mon 09 Nov 2015 08:51, taylanbayi...@gmail.com (Taylan Ulrich 
>> "Bayırlı/Kammer") writes:
>>
>>> The relevant bug report is .
>>>
>>> According to Ludo's explanation, compiling a module file leads to the
>>> module being created in the runtime, but with syntax bindings only, and
>>> runtime bindings missing.  That's what I mean with "degenerate" module
>>> for lack of a better term.  Loading the same file explicitly afterwards
>>> (or using load-compiled on the generated .go) seems to solve the issue.
>>
>> Ah, I see.  Well, one workaround for this issue, if you wanted to work
>> around it, would be to load the file before compiling it.  In that way
>> the module would already be loaded at the right phase.  This wouldn't
>> cost very much AFAIU.
>
> Yeah, loading before compiling also seems to work.

Just a note -- you should use `primitive-load' or otherwise ensure that
guile isn't doing auto-compiling.  `load' will auto-compile if it can.

A

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo  writes:

> On Mon 09 Nov 2015 08:51, taylanbayi...@gmail.com (Taylan Ulrich 
> "Bayırlı/Kammer") writes:
>
>> The relevant bug report is .
>>
>> According to Ludo's explanation, compiling a module file leads to the
>> module being created in the runtime, but with syntax bindings only, and
>> runtime bindings missing.  That's what I mean with "degenerate" module
>> for lack of a better term.  Loading the same file explicitly afterwards
>> (or using load-compiled on the generated .go) seems to solve the issue.
>
> Ah, I see.  Well, one workaround for this issue, if you wanted to work
> around it, would be to load the file before compiling it.  In that way
> the module would already be loaded at the right phase.  This wouldn't
> cost very much AFAIU.

Yeah, loading before compiling also seems to work.

Neither solves the other error I got, so I guess that one is really
unrelated to this.  I'm still clueless on that one.

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Mon 09 Nov 2015 08:51, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

>> I do not understand what you mean by "degenerate" modules :)  An
>> interpreted module should act the same as a compiled module.  I am
>> interested to hear what difference you can perceive between the two.
>
> The relevant bug report is .
>
> According to Ludo's explanation, compiling a module file leads to the
> module being created in the runtime, but with syntax bindings only, and
> runtime bindings missing.  That's what I mean with "degenerate" module
> for lack of a better term.  Loading the same file explicitly afterwards
> (or using load-compiled on the generated .go) seems to solve the issue.

Ah, I see.  Well, one workaround for this issue, if you wanted to work
around it, would be to load the file before compiling it.  In that way
the module would already be loaded at the right phase.  This wouldn't
cost very much AFAIU.

Andy

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo  writes:

> On Fri 06 Nov 2015 16:41, taylanbayi...@gmail.com (Taylan Ulrich 
> "Bayırlı/Kammer") writes:
>
>> Andy Wingo  writes:
>>
>>> On Thu 05 Nov 2015 17:10, taylanbayi...@gmail.com (Taylan Ulrich 
>>> "Bayırlı/Kammer") writes:
>>>
 It used to max out every CPU core, now just one. :-)
 It used to take ~18 minutes on my machine, now less than 3.
>>>
>>> If you compile within a par-for-each you should be able to peg your CPU
>>> core again, but actually reduce the time :)
>>
>> From what I understand, that would probably ignite the bug again.
>>
>> We need to ensure that as soon as a module file is compiled, it's also
>> explicitly loaded before anything else is compiled (which might import
>> it), otherwise that compilation will import the "degenerate" version of
>> the module that results from compiling but not loading it.
>>
>> But that's really just my shallow high-level understanding of the bug,
>> and could be way off.  If you have any insights on what's really going
>> on, that would be greatly appreciated. :-)
>
> AFAIU the problem that makes compilation slow is that *expansion* is
> slow.  When all your Scheme files are fresh, compiling 1 module has to
> expand all N modules.  Using one process to compile avoids this N^2
> penalty, just paying the O(N) cost up-front and then the marginal
> compilation cost is O(1).

Right, I was conflating expansion and compilation.

> There is benefit to compiling support modules before compiling (gnu
> packages) so that Guix's macros run compiled and not interpreted, but if
> you already have all of the modules expanded and loaded I don't think
> there is any advantage to loading the freshly compiled .go files.
>
> I do not understand what you mean by "degenerate" modules :)  An
> interpreted module should act the same as a compiled module.  I am
> interested to hear what difference you can perceive between the two.

The relevant bug report is .

According to Ludo's explanation, compiling a module file leads to the
module being created in the runtime, but with syntax bindings only, and
runtime bindings missing.  That's what I mean with "degenerate" module
for lack of a better term.  Loading the same file explicitly afterwards
(or using load-compiled on the generated .go) seems to solve the issue.

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Fri 06 Nov 2015 16:41, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

> Andy Wingo  writes:
>
>> On Thu 05 Nov 2015 17:10, taylanbayi...@gmail.com (Taylan Ulrich 
>> "Bayırlı/Kammer") writes:
>>
>>> It used to max out every CPU core, now just one. :-)
>>> It used to take ~18 minutes on my machine, now less than 3.
>>
>> If you compile within a par-for-each you should be able to peg your CPU
>> core again, but actually reduce the time :)
>
> From what I understand, that would probably ignite the bug again.
>
> We need to ensure that as soon as a module file is compiled, it's also
> explicitly loaded before anything else is compiled (which might import
> it), otherwise that compilation will import the "degenerate" version of
> the module that results from compiling but not loading it.
>
> But that's really just my shallow high-level understanding of the bug,
> and could be way off.  If you have any insights on what's really going
> on, that would be greatly appreciated. :-)

AFAIU the problem that makes compilation slow is that *expansion* is
slow.  When all your Scheme files are fresh, compiling 1 module has to
expand all N modules.  Using one process to compile avoids this N^2
penalty, just paying the O(N) cost up-front and then the marginal
compilation cost is O(1).

There is benefit to compiling support modules before compiling (gnu
packages) so that Guix's macros run compiled and not interpreted, but if
you already have all of the modules expanded and loaded I don't think
there is any advantage to loading the freshly compiled .go files.

I do not understand what you mean by "degenerate" modules :)  An
interpreted module should act the same as a compiled module.  I am
interested to hear what difference you can perceive between the two.

Andy

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> Could you do ‘guix pull --verbose’ so we see the order in which files
> are compiled?

Here's the output:

compiling...  0.0% of 463 files
compiling '/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix.scm'...
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
compiling...  0.2% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/base32.scm'...
compiling...  0.4% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/base64.scm'...
compiling...  0.6% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system.scm'...
compiling...  0.9% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/cmake.scm'...
compiling...  1.1% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/emacs.scm'...
compiling...  1.3% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/glib-or-gtk.scm'...
compiling...  1.5% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/gnu.scm'...
compiling...  1.7% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/haskell.scm'...
compiling...  1.9% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/perl.scm'...
compiling...  2.2% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/python.scm'...
compiling...  2.4% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/r.scm'...
compiling...  2.6% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/ruby.scm'...
compiling...  2.8% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/trivial.scm'...
compiling...  3.0% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build-system/waf.scm'...
compiling...  3.2% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build/cmake-build-system.scm'...
compiling...  3.5% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build/cvs.scm'...
compiling...  3.7% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build/download.scm'...
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; guix/build/download.scm:251:17: warning: possibly unbound variable 
`make-session'
;;; guix/build/download.scm:251:17: warning: possibly unbound variable 
`connection-end/client'
;;; guix/build/download.scm:259:8: warning: possibly unbound variable 
`set-session-server-name!'
;;; guix/build/download.scm:259:8: warning: possibly unbound variable 
`server-name-type/dns'
;;; guix/build/download.scm:263:4: warning: possibly unbound variable 
`set-session-transport-fd!'
;;; guix/build/download.scm:264:4: warning: possibly unbound variable 
`set-session-default-priority!'
;;; guix/build/download.scm:265:4: warning: possibly unbound variable 
`set-session-credentials!'
;;; guix/build/download.scm:265:38: warning: possibly unbound variable 
`make-certificate-credentials'
;;; guix/build/download.scm:271:4: warning: possibly unbound variable 
`handshake'
;;; guix/build/download.scm:272:18: warning: possibly unbound variable 
`session-record-port'
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload make-session in (gnutls):
;;; ERROR: missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; ERROR: missing interface for module (gnutls)
compiling...  3.9% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build/emacs-build-system.scm'...
compiling...  4.1% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/build/emacs-utils.scm'...
compiling...  4.3% of 463 files
compiling 
'/gnu/store/dbkpm8rqjz096w0bchmnlj8gw15q2riy-guix-latest/guix/bui

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Sadly, when I reintroduce sorting, I get a different error:
>
> === snip ===
> compiling...   17.9% of 463 filesBacktrace:
> In ice-9/boot-9.scm:
> 2401: 19 [save-module-excursion # ice-9/boot-9.scm:3066:17 ()>]
> 3085: 18 [#]
> In unknown file:
>?: 17 [primitive-load-path "gnu/packages/cross-base" ...]
> In ice-9/eval.scm:
>  505: 16 [# (define-module # 
> # ...)]
> In ice-9/psyntax.scm:
> 1106: 15 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
>  989: 14 [scan ((define-module (gnu packages cross-base) #:use-module ...)) 
> () ...]
>  279: 13 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
> In ice-9/eval.scm:
>  411: 12 [eval # ()]
> In ice-9/boot-9.scm:
> 2951: 11 [define-module* (gnu packages cross-base) #:filename ...]
> 2926: 10 [resolve-imports ((#) (#) (#) (#) ...)]
> 2864: 9 [resolve-interface (gnu packages commencement) #:select ...]
> 2789: 8 [# autoload version #:key ensure)> # ...]
> 3065: 7 [try-module-autoload (gnu packages commencement) #f]
> 2401: 6 [save-module-excursion # ice-9/boot-9.scm:3066:17 ()>]
> 3085: 5 [#]
> In unknown file:
>?: 4 [primitive-load-path "gnu/packages/commencement" ...]
> In ice-9/eval.scm:
>  432: 3 Exception thrown while printing backtrace:
> ERROR: In procedure package-location: Wrong type argument: Error while 
> printing exception.
>
> ice-9/boot-9.scm:106:20: In procedure # ice-9/boot-9.scm:97:6 (thrown-k . args)>:
> ice-9/boot-9.scm:106:20: In procedure package-version: Wrong type argument: 
> Error while printing exception.
> builder for `/gnu/store/94p9v7mgdl6n02skmbxnmmr82bxyz7bi-guix-latest.drv' 
> failed with exit code 1
> === / snip ===

Could you do ‘guix pull --verbose’ so we see the order in which files
are compiled?

TIA,
Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> We need to ensure that as soon as a module file is compiled, it's also
> explicitly loaded before anything else is compiled (which might import
> it), otherwise that compilation will import the "degenerate" version of
> the module that results from compiling but not loading it.

Correct.

Ludo’.

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo  writes:

> On Thu 05 Nov 2015 17:10, taylanbayi...@gmail.com (Taylan Ulrich 
> "Bayırlı/Kammer") writes:
>
>> CPU:
>>
>> It used to max out every CPU core, now just one. :-)
>>
>>
>> Time:
>>
>> It used to take ~18 minutes on my machine, now less than 3.
>
> If you compile within a par-for-each you should be able to peg your CPU
> core again, but actually reduce the time :)

>From what I understand, that would probably ignite the bug again.

We need to ensure that as soon as a module file is compiled, it's also
explicitly loaded before anything else is compiled (which might import
it), otherwise that compilation will import the "degenerate" version of
the module that results from compiling but not loading it.

But that's really just my shallow high-level understanding of the bug,
and could be way off.  If you have any insights on what's really going
on, that would be greatly appreciated. :-)

Taylan

Re: [PATCH] build: pull: Compile .scm files in one process.

[Andy Wingo]

On Thu 05 Nov 2015 17:10, taylanbayi...@gmail.com (Taylan Ulrich 
"Bayırlı/Kammer") writes:

> CPU:
>
> It used to max out every CPU core, now just one. :-)
>
>
> Time:
>
> It used to take ~18 minutes on my machine, now less than 3.

If you compile within a par-for-each you should be able to peg your CPU
core again, but actually reduce the time :)

Andy

Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

l...@gnu.org (Ludovic Courtès) writes:

> taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
>
>> It used to take ~18 minutes on my machine, now less than 3.
>
> Sounds compelling.  :-)
>
>> From fd5d9bf77fd38fad354d66c31e57bc9bbc86481f Mon Sep 17 00:00:00 2001
>> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>>  
>> Date: Thu, 5 Nov 2015 16:21:48 +0100
>> Subject: [PATCH] build: pull: Compile .scm files in one process.
>>
>> * guix/build/pull.scm (call-with-process): Removed procedure.
>> (report-build-progress): Removed procedure.
>> (p-for-each): Removed procedure.
>> (build-guix): Don't create subprocesses to compile the .scm files.
>
> [...]
>
>> -;; Build guix/*.scm before gnu/*.scm to speed
>> -;; things up.
>> -(sort (find-files out "\\.scm")
>> -  (let ((guix (string-append out "/guix"))
>> -(gnu  (string-append out "/gnu")))
>> -(lambda (a b)
>> -  (or (and (string-prefix? guix a)
>> -   (string-prefix? gnu b))
>> -  (string
> This logic should be kept (with this version of the patch, it starts
> compiling gnu.scm, which depends on many modules, so the progress report
> stays at 0% for maybe 30s and then suddenly reaches 20%.)

Ah, I see.  Indeed I was irritated by that long pause.

Sadly, when I reintroduce sorting, I get a different error:

=== snip ===
compiling... 17.9% of 463 filesBacktrace:
In ice-9/boot-9.scm:
2401: 19 [save-module-excursion #]
3085: 18 [#]
In unknown file:
   ?: 17 [primitive-load-path "gnu/packages/cross-base" ...]
In ice-9/eval.scm:
 505: 16 [# (define-module # # 
...)]
In ice-9/psyntax.scm:
1106: 15 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
 989: 14 [scan ((define-module (gnu packages cross-base) #:use-module ...)) () 
...]
 279: 13 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 12 [eval # ()]
In ice-9/boot-9.scm:
2951: 11 [define-module* (gnu packages cross-base) #:filename ...]
2926: 10 [resolve-imports ((#) (#) (#) (#) ...)]
2864: 9 [resolve-interface (gnu packages commencement) #:select ...]
2789: 8 [# # ...]
3065: 7 [try-module-autoload (gnu packages commencement) #f]
2401: 6 [save-module-excursion #]
3085: 5 [#]
In unknown file:
   ?: 4 [primitive-load-path "gnu/packages/commencement" ...]
In ice-9/eval.scm:
 432: 3 Exception thrown while printing backtrace:
ERROR: In procedure package-location: Wrong type argument: Error while printing 
exception.

ice-9/boot-9.scm:106:20: In procedure #:
ice-9/boot-9.scm:106:20: In procedure package-version: Wrong type argument: 
Error while printing exception.
builder for `/gnu/store/94p9v7mgdl6n02skmbxnmmr82bxyz7bi-guix-latest.drv' 
failed with exit code 1
=== / snip ===

This prompted me to try a different hack which is to first *only* load
all module files and later compile them, but that produced a similar
error:

=== snip ===
Backtrace:
In ice-9/boot-9.scm:
3065: 19 [try-module-autoload (gnu packages cross-base) #f]
2401: 18 [save-module-excursion #]
3085: 17 [#]
In unknown file:
   ?: 16 [primitive-load-path "gnu/packages/cross-base" ...]
In ice-9/eval.scm:
 505: 15 [# (define-module # # 
...)]
In ice-9/psyntax.scm:
1106: 14 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
 989: 13 [scan ((define-module (gnu packages cross-base) #:use-module ...)) () 
...]
 279: 12 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 11 [eval # ()]
In ice-9/boot-9.scm:
2951: 10 [define-module* (gnu packages cross-base) #:filename ...]
2926: 9 [resolve-imports ((#) (#) (#) (#) ...)]
2864: 8 [resolve-interface (gnu packages commencement) #:select ...]
2789: 7 [# # ...]
3065: 6 [try-module-autoload (gnu packages commencement) #f]
2401: 5 [save-module-excursion #]
3085: 4 [#]
In unknown file:
   ?: 3 [primitive-load-path "gnu/packages/commencement" ...]
In ice-9/eval.scm:
 432: 2 Exception thrown while printing backtrace:
ERROR: In procedure package-location: Wrong type argument: Error while printing 
exception.

ice-9/eval.scm:411:25: In procedure eval:
ice-9/eval.scm:411:25: In procedure package-version: Wrong type argument: Error 
while printing exception.
builder for `/gnu/store/cbd2vx8pbbhz098dfzwxj82n9h5lcvy4-guix-latest.drv' 
failed with exit code 1
=== / snip ===

I don't know if this relates to bug 15602, and I don't know how to
attempt to fix it.  Help appreciated.

>> +;; Compile the .scm files.  Also load every compiled file after writing 
>> it
>> +;; to work around  (FIXME).
>> +(let* ((files (filter (cut string-suffix? ".scm" <>)
>> +  (find-files out "\\.scm")))
>> +   (total (length files)))
>> +  (let loop ((file (car files))
>> +   

Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

taylanbayi...@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> It used to take ~18 minutes on my machine, now less than 3.

Sounds compelling.  :-)

> From fd5d9bf77fd38fad354d66c31e57bc9bbc86481f Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  
> Date: Thu, 5 Nov 2015 16:21:48 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process): Removed procedure.
> (report-build-progress): Removed procedure.
> (p-for-each): Removed procedure.
> (build-guix): Don't create subprocesses to compile the .scm files.

[...]

> -;; Build guix/*.scm before gnu/*.scm to speed
> -;; things up.
> -(sort (find-files out "\\.scm")
> -  (let ((guix (string-append out "/guix"))
> -(gnu  (string-append out "/gnu")))
> -(lambda (a b)
> -  (or (and (string-prefix? guix a)
> -   (string-prefix? gnu b))
> -  (string +;; Compile the .scm files.  Also load every compiled file after writing 
> it
> +;; to work around  (FIXME).
> +(let* ((files (filter (cut string-suffix? ".scm" <>)
> +  (find-files out "\\.scm")))
> +   (total (length files)))
> +  (let loop ((file (car files))
> + (files (cdr files))
> + (completed 0))
> +(display #\cr log-port)
> +(format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
> +(* 100. (/ completed total)) total)
> +(force-output log-port)
> +(let ((go (string-append (string-drop-right file 4)
> + ".go")))
> +  (format debug-port "~%compiling '~a'...~%" file)
> +  (parameterize ((current-warning-port debug-port))
> +(compile-file file
> +  #:output-file go
> +  #:opts
> +  %auto-compilation-options)
> +(load-compiled go)))

So this simple ‘load-compiled’ call addresses
, right?  This is where the comment and
explanation about this bug should go.

The ‘load-compiled’ code seems to be the only difference with 52af657a,
which was right before we noticed #15602 (which led to 178f77b2.)

Another stylistic issue: please use ‘match’ instead of the unnameable
primitives.  ;-)

Could you send an updated patch?

Thanks!

Ludo’.