Re: 11/12: gnu: Add libuemf.

[Caleb Ristvedt]

Maxim Cournoyer  writes:
> On the current master branch,
>
> make check TESTS=tests/guix-package.sh
>
> [...]
>
> PASS: tests/guix-package.sh
> 
> Testsuite summary for GNU Guix 1.0.1.17120-e7b86a0
> 
> # TOTAL: 1
> # PASS:  1
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
> # ERROR: 0
> 
>
> 'guix lint libuemf' also passes, and 'guix show libuemf' renders the
> description correctly here.
>
> Maxim

Indeed, Tobias worked around it in
86a03090afa711dfa2f141caee820b66d7942bc3.

- reepca


signature.asc
Description: PGP signature

Re: 11/12: gnu: Add libuemf.

[Caleb Ristvedt]

guix-comm...@gnu.org writes:

> apteryx pushed a commit to branch master
> in repository guix.
>
> commit 3f1f98d9d8c4e7f5dfca921fe1957e4d53783e35
> Author: Maxim Cournoyer 
> AuthorDate: Tue Jan 28 01:21:24 2020 -0500
>
> gnu: Add libuemf.
> 
> * gnu/packages/image.scm (libuemf): New variable.
> ---
>  gnu/packages/image.scm | 51 
> ++
>  1 file changed, 51 insertions(+)
>
> diff --git a/gnu/packages/image.scm b/gnu/packages/image.scm
> index 24f1b15..05b64dd 100644
> --- a/gnu/packages/image.scm
> +++ b/gnu/packages/image.scm
> @@ -25,6 +25,7 @@
>  ;;; Copyright © 2018 Rutger Helling 
>  ;;; Copyright © 2020 Giacomo Leidi 
>  ;;; Copyright © 2020 R Veera Kumar 
> +;;; Copyright © 2020 Maxim Cournoyer 
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;

<...>

> +(description "The libUEMF library is a portable C99 implementation for
> +reading and writing @abbr{WFM, Windows Metafile}, @abbr{EMF, Enhanced
> +Metafile}, and @abbr{EMF+, Enhanced Metafile Plus} files.")
> +(license license:gpl2+)))

Turns out guile's texinfo stuff chokes on @abbr:


$ guix package -s libuemf
Backtrace:
   1 (primitive-load "/home/reepca/.config/guix/current/bin/guix")
In guix/ui.scm:
  1936:12  0 (run-guix-command _ . _)

guix/ui.scm:1936:12: In procedure run-guix-command:
Throw to key `parser-error' with args `(#f "Unknown command" abbr)'.


guix/ui.scm:1936:12: In procedure run-guix-command:
Throw to key `parser-error' with args `(#f "Unknown command" abbr)'.

I don't know enough about texinfo to offer a solution, but figured it
should be mentioned. Currently this is causing tests/packages.sh to
fail.

- reepca

Re: Packaging Jami progress

[Caleb Ristvedt]

Jan Wielkiewicz  writes:

> I tried both ways - the second works, but the first doesn't.

That would be the "in theory, it would work" part. On further investigation,
source-module-closure has a #:select? keyword argument, which takes a module
name and returns #f if it shouldn't be included in the closure. By default it's
'guix-module-name?', so it only includes the guix modules, and not, for example,
(gcrypt hash). One might try passing #:select? (const #t), but this would likely
reveal further issues - for example, guile-gcrypt doesn't work without
libgcrypt, but source-module-closure isn't going to pull that in.

The short answer is "yeah, for big closures that reach outside of guix (or
especially that have non-scheme dependencies) source-module-closure isn't
perfect", and build-side code should try to minimize the size of the closure it
pulls in (which, for pretty much any (gnu packages ...) module, is going to be
huge). The second solution is probably the better one here.

There's this sort of awkward middle ground we don't see much where a
build-side procedure has to be specific to some relatively small set of
packages, but still to enough packages that repeating it in the builder
for each of those packages duplicates a lot of code. Splicing the
definition into the builder programmatically is a bit of a hack, as it's
still duplicated between each builder interned in the store, but much
better than copy+pasting manually.

Hope that helps.

- reepca

Re: Packaging Jami progress

[Caleb Ristvedt]

#:modules and #:imported-modules are distinct arguments. #:modules is the
modules that your builder is going to use (as in "they go in a (use-modules
...) form"), while #:imported-modules is the modules that need to be
available
in the build environment. It's complaining at build-time that it can't find
that
module to use, because you haven't told it to include that module in the
build
environment. #:imported-modules should give a superset of what #:modules
gives,
especially if a module in use is going to have indirect
requirements. Thankfully, wrangling together those indirect requirements is
already implemented in (guix modules) as source-module-closure.

Thus, you could conceptually do


(arguments
 `(#:imported-modules (,@(source-module-closure
  '((gnu packages jami)
,@%gnu-build-system-modules)))
   #:modules ((gnu packages jami)
  ,@(@@ (guix build-system gnu) %default-modules

And in theory it would work. Note, though, that this would pull in the
entire
module dependency graph of (gnu packages jami), and this may include things
that
source-module-closure would have a hard time detecting and aren't really
needed. Ideally this procedure would be generalized and put in (guix build
), but I can understand if that's not possible. Note also that
you
could simply splice in the definition of your procedure into the builder
manually, like so:

(define my-procedure-code '(lambda (a b c) ...))

(arguments
 `(#:phases (let ((my-procedure ,my-procedure-code)) (modify-phases ...

Hope that helps, thanks for the work on Jami y'all!

On Tue, Dec 10, 2019 at 8:57 AM Pierre Neidhardt  wrote:

> Sorry, I don't know how to import a procedure defined in the same file :(
> Anyone?
>
> --
> Pierre Neidhardt
> https://ambrevar.xyz/
>

Re: can't get past commencement in test-env

[Caleb Ristvedt]

Maxim Cournoyer  writes:

> Hello Caleb,
>
> Sorry if this answer is late; I noticed nobody had replied yet, so
> thought I'd try.

Thanks.

> That's an interesting find! What exactly is the 'test-env'? Do you mean
> an environment created using 'guix environment --pure' ?

No, test-env is the name of a script generated in the process of
compiling guix. It's used in running the tests run by 'make check'. The
main feature is that it runs its own daemon with its own store in a
subdirectory of the guix source directory named test-tmp/store, so tests
involving the daemon can be run without needing it already
installed. Other than that, it works like pre-inst-env for the most
part.

>> What is The Right Thing™ to do here?
>
> Perhaps the best fix would be to find a way to tell gcc to always use
> "/lib" for its library directory?  Perhaps possible via a configure
> flag?  Otherwise by patching the script.

genmultilib takes many arguments, passed to it from the makefile. I'm
sure there's a combination of values for which it will do what we
currently do. To work correctly, though, the script will definitely have
to be patched so that the script it generates has the correct
interpreter.

> Until we have a need for separating /lib and /lib64 (I guess if we
> wanted to produce 32 bits variants of packages, that could be useful),
> we can postpone the big change of using the "correct" library path.

Aye. Changing the builder for gcc-boot0 will cause a full-world rebuild,
so I suppose it should go in core-updates?

- reepca

can't get past commencement in test-env

[Caleb Ristvedt]

gcc-boot0 in (gnu packages commencement) compiles subtly differently
when built in a chroot (for example, by an installed daemon) compared to
when built without root privileges (for example, in
test-env). Specifically, the presence of this line in the build log:

../../gcc-5.5.0/gcc/genmultilib: ./tmpmultilib: /bin/sh: bad interpreter: No 
such file or directory

This doesn't get caught by the patch-shebangs or
patch-generated-shebangs phases because tmpmultilib is both generated
and immediately executed by genmultilib in order to, I kid you not,
implement recursion in a portable manner by having tmpmultilib run
itself. Somehow this works out in the chroot case despite it failing to
run, but in the non-chroot case /bin/sh actually exists, at least on
Guix System. This ultimately causes two different compilers to be
created in the two cases. In the chroot case, 'g++
-print-multi-os-directory' simply gives

.;

while in the non-chroot case, it gives

../lib64

This means that when the bootstrap continues, libstdc++ is built with
its outputs going to different locations (lib/ or lib64/) depending on
which of the two compilers builds it. gcc-final assumes it will be in
lib, and so it can't find it if libstdc++ was built with a gcc-boot0
built without a chroot.

I don't know much about multilib stuff, but it would seem that the
"correct" output from gcc's perspective would be one in which its
contemplation of the matter isn't interrupted by a "bad interpreter"
error. But that's also the opposite of the assumptions we currently
make, and changing it would require both rebuilding the world and
modifying several packages.

At the same time, I'd like to be able to test building derivations in
test-env without needing to run it as root (and modifying test-env
slightly to remove the --disable-chroot, and choosing between running
all those builders as root (yikes) or risking interfering with my
installed daemon by using the same build group). I'd also appreciate it
if others could test building packages in test-env easily, as it catches
an entire class of errors that usually isn't caught otherwise (typically
errors caused by assumptions about where the store is). The same issues
that plague test-env will also occur in an unprivileged install.

What is The Right Thing™ to do here?

- reepca

add-to-store, add-text-to-store, and store path hash parts

[Caleb Ristvedt]

I'm currently focusing on two particular RPCs that we use, addToStore
and addTextToStore. In the Nix thesis I only see reference to
addToStore. What's the difference between the two? I see that
addTextToStore allows specifying references while addToStore doesn't,
and addTextToStore is specifically for regular files. What's the
motivation for the split? Additionally, the hash part of the resulting
path is computed differently for addTextToStore, addToStore, and
addToStore with recursive set. This means that there could be three
regular files with the exact same contents and the exact same name, but
different hash parts. It doesn't violate any store invariants or
anything, of course, but I'm wondering why it's done.

Especially puzzling to me is what the purpose of the recursive flag is
for addToStore. It causes the path hash to be calculated differently and
makes sure that non-regular files cause errors, but why?

Thanks in advance for any answers y'all can give.

- reepca

Re: 02/09: guix: store: Make register-items transactional, register drv outputs

[Caleb Ristvedt]

> I finally got around to fixing it in
> a31174e896047e6a0f42b69db331fdeebb3cc995.
>
> The kludge is no longer needed!

Great. Here are updated patches:

>From 287879a825f41c46cc5091c715467e476d465def Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Mon, 1 Apr 2019 15:04:59 -0500
Subject: [PATCH 1/2] guix: split (guix store) and (guix derivations).

* guix/store.scm (%store-prefix, store-path, output-path, fixed-output-path,
  store-path?, direct-store-path?, derivation-path?, store-path-package-name,
  store-path-hash-part, direct-store-path, derivation-log-file): Moved
  to (guix store files) and re-exported from here.
  ((guix store files)): use it.

* guix/store/files.scm: new module.
  above named variables: added.

* guix/derivations.scm (, derivation?, derivation-outputs,
  derivation-inputs, derivation-sources, derivation-system,
  derivation-builder, derivation-builder-arguments,
  derivation-builder-environment-vars, derivation-file-name,
  derivation-output>, derivation-output?, derivation-output-path,
  derivation-output-hash-algo, derivation-output-hash,
  derivation-output-recursive?, derivation-input>, derivation-input?,
  derivation-input-path, derivation-input-sub-derivations, read-derivation,
  read-derivation-from-file, write-derivation): Moved to (guix store
  derivations) and re-exported from here.
  ((guix store derivations)): use it.

* guix/store/derivations.scm: new module.
  above named variables: added.
---
 guix/derivations.scm   | 281 
 guix/store.scm | 155 ++--
 guix/store/derivations.scm | 287 +
 guix/store/files.scm   | 171 ++
 4 files changed, 502 insertions(+), 392 deletions(-)
 create mode 100644 guix/store/derivations.scm
 create mode 100644 guix/store/files.scm

diff --git a/guix/derivations.scm b/guix/derivations.scm
index fb2fa177be..483b274e53 100644
--- a/guix/derivations.scm
+++ b/guix/derivations.scm
@@ -39,31 +39,10 @@
   #:use-module (guix base32)
   #:use-module (guix records)
   #:use-module (guix sets)
-  #:export (
-derivation?
-derivation-outputs
-derivation-inputs
-derivation-sources
-derivation-system
-derivation-builder
-derivation-builder-arguments
-derivation-builder-environment-vars
-derivation-file-name
+  #:use-module (guix store derivations)
+  #:export (derivation-input-output-paths
 derivation-prerequisites
 derivation-prerequisites-to-build
-
-
-derivation-output?
-derivation-output-path
-derivation-output-hash-algo
-derivation-output-hash
-derivation-output-recursive?
-
-
-derivation-input?
-derivation-input-path
-derivation-input-sub-derivations
-derivation-input-output-paths
 valid-derivation-input?
 
 &derivation-error
@@ -82,9 +61,6 @@
 derivation-hash
 derivation-properties
 
-read-derivation
-read-derivation-from-file
-write-derivation
 derivation->output-path
 derivation->output-paths
 derivation-path->output-path
@@ -107,7 +83,33 @@
 build-expression->derivation)
 
   ;; Re-export it from here for backward compatibility.
-  #:re-export (%guile-for-build))
+  #:re-export (%guile-for-build
+   
+   derivation?
+   derivation-outputs
+   derivation-inputs
+   derivation-sources
+   derivation-system
+   derivation-builder
+   derivation-builder-arguments
+   derivation-builder-environment-vars
+   derivation-file-name
+
+   
+   derivation-output?
+   derivation-output-path
+   derivation-output-hash-algo
+   derivation-output-hash
+   derivation-output-recursive?
+
+   
+   derivation-input?
+   derivation-input-path
+   derivation-input-sub-derivations
+
+   read-derivation
+   read-derivation-from-file
+   write-derivation))
 
 ;;;
 ;;; Error conditions.
@@ -121,48 +123,6 @@
   derivation-missing-output-error?
   (output derivation-missing-output))
 
-;;;
-;;; Nix derivations, as implemented in Nix's `derivations.cc'.
-;;;
-
-(define-immutable-record-type 
-  (make-derivation outputs inputs sources system builder args env-vars
-   file-name)
-  derivation?
-  (outputs  derivation-outputs)  ; list of name/ pairs
-  (inputs   derivation-inputs)   ; list of 
-  (sources  derivation-sources)  ; list of store paths
-  (system   derivation-system)   ; string
-  (builder 

Re: 02/09: guix: store: Make register-items transactional, register drv outputs

[Caleb Ristvedt]

> A problem is that we should not pull in these two modules here, because
> they are conceptually at a higher level (they have to do with talking to
> a separate daemon process.)
>
> So perhaps a first step would be to re-arrange things, probably along
> these lines:
>
>   • Move the .drv parsing code and the  record type from
> (guix derivations) to, say, (guix store derivations).
>
>   • Re-export all these bindings from (guix derivations).
>
>   • Move ‘store-path?’, ‘derivation-path?’ & co. (everything below line
> 1745 in guix/store.scm) to, say, (guix store files).  Re-export
> appropriately so the API remains unchanged.
>
> At this point, (guix store …) modules won’t have to use (guix store) and
> (guix derivations) at all.
>
> How does that sound?

As we found out on IRC yesterday, https://bugs.gnu.org/15602 is causing
this to fail. If I understand correctly, the workaround involves trying
to make sure that modules that use (guix memoization) are compiled
before (guix memoization) is (and it turns out that once that's solved
the same issue occurs with (guix config)). Considering that the
module-import-compiled builders just do a depth-first-search for scheme
source files ordered alphabetically, that's a bit tricky. I ended up
making a dummy module, (gnu build dummy), that does nothing but use
(guix config) and (guix memoization), and the tests now pass.

Does a better workaround exist? Could we pass an ordered list of files
to the builder somehow instead of just a directory? But then again,
changing every module-import-compiled would mean rebuilding the
world. Eh.

Thoughts?

Oh, also, should I just copy all the copyright lines from the old
modules to the new ones?

- reepca

Re: 02/09: guix: store: Make register-items transactional, register drv outputs

[Caleb Ristvedt]

Changes made, though I'm not quite sure about this part:

> Could you add a test in tests/store-database.scm for this bit?
>
>> +(when (derivation-path? real-file-name)
>> +  (register-derivation-outputs))

Should that be a separate test or an extension of "register-path"?
Currently I'm doing the latter. And it looks ugly. The C++ code also
checks the validity of derivation outputs (whether the outputs should
actually have those filenames), which I'd like to eventually add to
register-items, and it would cause an error with the way the test
currently is. But then again, ff...fff-fake is probably already not
the right filename to be generated for that file anyway. Should we try
to do things "the proper way" there?

> Could you send updated patches?

>From 5ae8c31826f06f4ad0b52a4d7b0cd6c4abc64a20 Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Wed, 30 Jan 2019 17:03:38 -0600
Subject: [PATCH 1/2] guix: store: Make register-items transactional.

* guix/store/database.scm (SQLITE_BUSY, register-output-sql): new variables
  (add-references): don't try finalizing after each use, only after all the
  uses (otherwise a finalized statement would be used if #:cache? was #f).
  (call-with-transaction): New procedure.
  (register-items): Use call-with-transaction to prevent broken intermediate
  states from being visible.

* .dir-locals.el (call-with-transaction): indent it.
---
 .dir-locals.el  |  1 +
 guix/store/database.scm | 50 -
 2 files changed, 40 insertions(+), 11 deletions(-)

diff --git a/.dir-locals.el b/.dir-locals.el
index 593c767d2b..550e06ef09 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -79,6 +79,7 @@
(eval . (put 'with-extensions 'scheme-indent-function 1))
 
(eval . (put 'with-database 'scheme-indent-function 2))
+   (eval . (put 'call-with-transaction 'scheme-indent-function 2))
 
(eval . (put 'call-with-container 'scheme-indent-function 1))
(eval . (put 'container-excursion 'scheme-indent-function 1))
diff --git a/guix/store/database.scm b/guix/store/database.scm
index 4791f49865..af7f82b049 100644
--- a/guix/store/database.scm
+++ b/guix/store/database.scm
@@ -96,6 +96,31 @@ create it and initialize it as a new database."
   (lambda ()
 (sqlite-close db)
 
+;; XXX: missing in guile-sqlite3@0.1.0
+(define SQLITE_BUSY 5)
+
+(define (call-with-transaction db proc)
+  "Start a transaction with DB (make as many attempts as necessary) and run
+PROC.  If PROC exits abnormally, abort the transaction, otherwise commit the
+transaction after it finishes."
+  (catch 'sqlite-error
+(lambda ()
+  ;; We use begin immediate here so that if we need to retry, we
+  ;; figure that out immediately rather than because some SQLITE_BUSY
+  ;; exception gets thrown partway through PROC - in which case the
+  ;; part already executed (which may contain side-effects!) would be
+  ;; executed again for every retry.
+  (sqlite-exec db "begin immediate;")
+  (let ((result (proc)))
+(sqlite-exec db "commit;")
+result))
+(lambda (key who error description)
+  (if (= error SQLITE_BUSY)
+  (call-with-transaction db proc)
+  (begin
+(sqlite-exec db "rollback;")
+(throw 'sqlite-error who error description))
+
 (define %default-database-file
   ;; Default location of the store database.
   (string-append %store-database-directory "/db.sqlite"))
@@ -172,9 +197,9 @@ ids of items referred to."
 (sqlite-bind-arguments stmt #:referrer referrer
#:reference reference)
 (sqlite-fold cons '() stmt)   ;execute it
-(sqlite-finalize stmt)
 (last-insert-row-id db))
-  references)))
+  references)
+(sqlite-finalize stmt)))
 
 (define* (sqlite-register db #:key path (references '())
   deriver hash nar-size time)
@@ -305,6 +330,7 @@ Write a progress report to LOG-PORT."
 (define real-file-name
   (string-append store-dir "/" (basename (store-info-item item
 
+
 ;; When TO-REGISTER is already registered, skip it.  This makes a
 ;; significant differences when 'register-closures' is called
 ;; consecutively for overlapping closures such as 'system' and 'bootcfg'.
@@ -325,12 +351,14 @@ Write a progress report to LOG-PORT."
   (mkdir-p db-dir)
   (parameterize ((sql-schema schema))
 (with-database (string-append db-dir "/db.sqlite") db
-  (let* ((prefix   (format #f "registering ~a items" (length items)))
- (progress (progress-reporter/bar (lengt

Re: guix gc takes long in "deleting unused links ..."

[Caleb Ristvedt]

Ludovic Courtès  writes:

> Note that the database would need to contain hashes of individual files,
> not just store items (it already contains hashes of store item nars).

Indeed! Although last night I thought of a way that it would only need
to contain hashes of *unique* individual files, see below.

> This issue was discussed a while back at
> .  Back then we couldn’t agree on
> a solution, but it’d be good to have your opinion with your fresh mind!

The main thing that comes to mind is making the amount of time required
for deleting links scale with the number of things being deleted rather
than the number of "things" in total - O(m) instead of O(n), so to
speak. I actually hadn't even considered things like disk access
patterns.

In my mind, the ideal situation is like this: we get rid of .links, and
instead keep a mapping from hashes to inodes in the
database. Deduplication would then involve just creating a hardlink to
the corresponding inode. The link-deleting phase then becomes entirely
unnecessary, as when the last hardlink is deleted the refcount becomes 0
automatically. Unfortunately, this isn't possible, because AFAIK there
is no way to create a hardlink to an inode directly; it always has to go
through another hardlink. Presumably the necessary system call doesn't
exist because there would be permissions / validation issues (if anyone
happens to know of a function that does something like this, I'd love to
hear about it!).

So the second-most-ideal situation would, to me, be to keep a mapping
from inodes to hashes in the database. Then, when it becomes known
during garbage collection that a file is to be deleted and the refcount
for its inode is 2, the file's inode can be obtained from stat(), from
that the hash can be looked up, and from that the corresponding link
filename can be obtained and removed. After that the inode->hash
association can be removed from the database.

I think this is a reasonable approach, as such a table in the database
shouldn't take up much more disk space than .links does: 8 bytes for an
inode, and 32 bytes for the hash (or 52 if we keep the hash in text
form), for a total of 60 bytes. Based on the numbers from the linked
discussion (10M entries), that's around 400MB or 600MB, plus whatever
extra space sqlite uses, kept on the disk. If that's considered too
high, we could only store the hashes of relatively large files in the
database and fall back to hashing at delete-time for the others.

The main limitation is the lack of portability of inodes. That is, when
copying a store across filesystems, said table would need to be
updated. Also, it requires that everything in the store is on the same
filesystem, though this could be fixed by looking up the hash through
(inode, device number) pairs instead of just inode. In that case, it
would work to copy across filesystems, though I think it still wouldn't
work for copying across systems.

How does that sound?


> The guix-daemon child that handles the session would immediately get
> SIGHUP and terminate (I think), but that’s fine: it’s just that files
> that could have been removed from .links will still be there.

Turns out it's SIGPOLL, actually, but yep. There's a checkInterrupt()
that gets run before each attempt to delete a link, and that triggers
the exit.

- reepca

Re: guix gc takes long in "deleting unused links ..."

[Caleb Ristvedt]

Björn Höfling  writes:

> Why does that take soo long?

Warning: technical overview follows.

It takes so long because after the garbage collection pass it then does
a *full* pass over the /gnu/store/.links directory. Which is huge. It
contains an entry for every unique file (not just store entry, but
everything in those entries, recursively) in the store. The individual
work for each entry is low - just a readdir(), lstat() to see if the
link is still in use anywhere, and an unlink() if it isn't. But my
store, for example, has 998536 entries in there. I got that number with
a combination of ls and wc, and it took probably around 4 minutes to get
it.

Ideally, the reference-counting approach to removing files would work
the same as in programming languages: as soon as a reference is removed,
check whether the reference count is now 0 (in our case 1, since an
entry would still exist in .links). In our case, we'd actually have to
check prior to losing the reference whether the count *would become* 1,
that is, whether it is currently 2. But unlike in programming languages,
we can't just "free a file" (more specifically, an inode). We have to
delete the last existing reference, in .links. The only way to find that
is by hashing the file prior to deleting it, which could be quite
expensive, but for any garbage collection targeting a small subset of
store items it would likely still be much faster. A potential fix there
would be to augment the store database with a table mapping store paths
to hashes (hashes already get computed when store items are
registered). Or we could switch between the full-pass and incremental
approaches based on characteristics of the request.

> Or better: Is it save here to just hit CTRL-C (and let the daemon work
> in background, or whatever)?

I expect that CTRL-C at that point would cause the guix process to
terminate, closing its connection to the daemon. I don't believe the
daemon uses asynchronous I/O, so it wouldn't be affected until it tried
reading or writing from/to that socket. So yeah, if you do that at that
point it would probably work, but you may as well just start it in the
background in that case ("guix gc ... &") or put it in the background
with CTRL-Z followed by the 'bg' command.

- reepca

Re: 02/19: guix: register-path: Implement prototype in scheme.

[Caleb Ristvedt]

Hello!

Ludovic Courtès  writes:

> Could you try and rebase ‘guile-daemon’ on top of these modules?  I’m
> happy to answer any questions you may have regarding modifications that
> I made.

I'm not sure what is meant by "on top of these modules". If I understand
correctly, the problem is that while technically guile-daemon is rebased
on master (ran the git commands, they finished succesfully, etc), it's
not really as if the changes were written "with master in mind", so to
speak. That is, we end up with weird-looking stuff like adding
register-path to (guix store) *after* we already have register-items,
just to remove it a few commits later, adding and then removing (guix
sql), etc. And while technically it ends up "in the right place" now, a
coherent commit history is always good.

Is that about right?

> Thanks for picking up work on the daemon!

I'd hate for all those hours poring over C++ to go wasted!

- reepca

Re: ‘sudo’ leaves PATH unchanged… so what?

[Caleb Ristvedt]

To be clear, it's not so much a real problem as an eyesore. 'sudo guix
package ...' and 'sudo guix pull' will operate on root's profile(s) as
expected, but that also means 'sudo guix system reconfigure' will look at
root's profile when deciding whether to say that it's out of date, even
though the guix actually used for reconfiguring may not be out of date. It
pretty much only ever comes up when reconfiguring, but for those like
myself who are worried by messages like that it can be confusing to try to
resolve.

On Fri, Jan 18, 2019 at 11:33 AM Giovanni Biscuolo  wrote:

> Hi all,
>
> Caleb Ristvedt  writes:
>
> > I'd just like to add that if a user has guix installed for root but only
> > really keeps their user's guix up to date (I imagine a fairly common
> > situation), they're in for a weird situation when using sudo: a
> > bleeding-edge guix will complain about being outdated, since sudo (even
> > with -E) sets $USER, which is used to determine which file's timestamp
> > should be used for deciding whether the installed guix is outdated.
> > Basically, your shiny new guix warns you that someone else's dirty old
> guix
> > is old.
>
> if this is true (root and user profiles using different versions of guix
> is a problem), should this be clearly stated in the manual?
>
> maybe I'm missing something, but I thought root and user profiles were
> isolated so I can have different binaries (guix included) and use them
> independently, am I wrong?
>
> Thanks!
> Giovanni
>
> [...]
>
> --
> Giovanni Biscuolo
>
> Xelera IT Infrastructures
>

Re: ‘sudo’ leaves PATH unchanged… so what?

[Caleb Ristvedt]

I'd just like to add that if a user has guix installed for root but only
really keeps their user's guix up to date (I imagine a fairly common
situation), they're in for a weird situation when using sudo: a
bleeding-edge guix will complain about being outdated, since sudo (even
with -E) sets $USER, which is used to determine which file's timestamp
should be used for deciding whether the installed guix is outdated.
Basically, your shiny new guix warns you that someone else's dirty old guix
is old.

On Fri, Jan 18, 2019 at 7:31 AM Ricardo Wurmus  wrote:

>
> Giovanni Biscuolo  writes:
>
> > IMHO it should be explicitly stated that in general it's not
> > advised/needed to install guix for the root user and that "guix system"
> > must be run as a normal user via sudo (and the user must be allowed to
> > do so by root via sudoers); then the example above
>
> Currently, installing Guix for the root user is what the installer
> script does and what the instructions for the binary installation method
> suggest.
>
> --
> Ricardo
>
>
>

hard-coded /gnu/store patches cause test-env to not work properly

[Caleb Ristvedt]

Well, I finally (it only took 4 days!) found out what was causing
python2-flake8@3.5.0 to fail to build only in the test environment:
python-2.7-site-prefixes.patch


Add all /gnu/store/ prefixes found in PYTHONPATH to the prefixes where
site-packages (and .pth files) are searched.

*** Python-2.7.11/Lib/site.py.orig  2016-10-17 23:27:23.746149690 +0200
--- Python-2.7.11/Lib/site.py   2016-10-17 23:44:51.930871644 +0200
***
*** 65,70 
--- 65,82 
  
  # Prefixes for site-packages; add additional prefixes like /usr/local here
  PREFIXES = [sys.prefix, sys.exec_prefix]
+ # Guix: Add all /gnu/store-paths in PYTHONPATH--these are all
+ # "prefixes".  This is required to search .pth files in all python
+ # packages contained in /gnu/store which is required to make
+ # .pth-defined namespace packages work.
+ # This is necessary if the packages are not merged into a single
+ # `site-packages` directory (like when using `guix environment`) but
+ # listed in PYTHONPATH (like when running `guix build`).
+ for p in sys.path:
+ if p.startswith('/gnu/store/'):
+ PREFIXES.append(p[:p.find('/', 44)]) # find first pathsep after hash
+ del p
+ 
  # Enable per user site-packages directory
  # set it to False to disable the feature or True to force the feature
  ENABLE_USER_SITE = None


Of course, /home/reepca/Programming/guix/test-tmp/store/ wouldn't be
recognized. I manually modified the patch and re-attempted building
flake8 with the new python2 and sure enough, it succeeded.

Perhaps more practically, this is also an issue for anyone who installed
guix with their store in a non-standard location. Are we currently
assuming that anyone installing in a non-/gnu/store location is going to
have NIX_STORE set? If so, the solution seems pretty easy: do what the
other patches involving /gnu/store do and first attempt to honor that,
only falling back to /gnu/store as a default. In that case, we should
also modify proot-test-fhs.patch, as it also hardcodes /gnu/store
without first attempting to honor NIX_STORE. This should at least
resolve the issue for test-env, since NIX_STORE is always set in
test-env and you don't need to run the programs there outside of
test-env in order to test building (the main purpose).

If we aren't assuming NIX_STORE is set for non-/gnu/store installs, then
I guess we'd have to start generating patches based on the configured
storedir. But I get the feeling it's not much of a concern.

Attached is a patch that fixes python-2.7-site-prefixes.patch.



signature.asc
Description: PGP signature
>From e9fdaec83568d14b5462757b473ed6f25b3f114e Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Tue, 8 Jan 2019 01:26:59 -0600
Subject: [PATCH] patches: honor NIX_STORE in site.py.

Previously various python packages would fail to work unless the store they
were kept in was /gnu/store. This fixes that.

* gnu/packages/patches/python-2.7-site-prefixes.patch: Try NIX_STORE first
  and only use /gnu/store as a fallback.
---
 gnu/packages/patches/python-2.7-site-prefixes.patch | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/patches/python-2.7-site-prefixes.patch b/gnu/packages/patches/python-2.7-site-prefixes.patch
index 9e3066508..0451d07f4 100644
--- a/gnu/packages/patches/python-2.7-site-prefixes.patch
+++ b/gnu/packages/patches/python-2.7-site-prefixes.patch
@@ -5,7 +5,7 @@ site-packages (and .pth files) are searched.
 --- Python-2.7.11/Lib/site.py	2016-10-17 23:44:51.930871644 +0200
 ***
 *** 65,70 
 65,82 
+--- 65,85 
   
   # Prefixes for site-packages; add additional prefixes like /usr/local here
   PREFIXES = [sys.prefix, sys.exec_prefix]
@@ -16,9 +16,12 @@ site-packages (and .pth files) are searched.
 + # This is necessary if the packages are not merged into a single
 + # `site-packages` directory (like when using `guix environment`) but
 + # listed in PYTHONPATH (like when running `guix build`).
++ guix_store = os.getenv("NIX_STORE")
++ if not guix_store:
++ guix_store = '/gnu/store'
 + for p in sys.path:
-+ if p.startswith('/gnu/store/'):
-+ PREFIXES.append(p[:p.find('/', 44)]) # find first pathsep after hash
++ if p.startswith(guix_store):
++ PREFIXES.append(p[:p.find('/', 34 + len(guix_store))]) # find first pathsep after hash
 + del p
 + 
   # Enable per user site-packages directory
-- 
2.20.0

Re: GSoC-2018

[Caleb Ristvedt]

Hi Sandeep/uniq10, welcome!

I'm the person that worked on the rewrite last year (I like to go by
reepca in conversations and irc). I'm glad that you'll be working on it
this summer. I still feel like I didn't do enough last summer, though,
so if there's any way I can help, I'd be glad to (though the next couple
of days are finals here, so I'll be rather busy just then).

If you have any questions about the code or find yourself reading
through a bunch of C++ you'd rather not be reading through and thinking
to yourself "that other guy probably already had to do this...", feel
free to ask. I'm in #guix pretty much all the time (though only on my
desktop, so if I'm away the response may be delayed), though you'll
probably have to ping me.

- reepca

GSoC final update

[Caleb Ristvedt]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

As the official work time for GSoC draws to a close, it seems fitting to
summarize where the project is at right now. *deep breath*

What works: building hello. I tried building SBCL as well, but building
one of the dependencies caused hanging (but the same thing happened when
I tried building it with the C++ daemon). In short, for at least simple
derivations, building derivations works.

What doesn't: Way too much. RPCs aren't implemented (for testing
building I've been manually running builds from a REPL within test-env),
scheduling of multiple concurrent derivations isn't implemented, garbage
collection isn't implemented, and a lot of features even of
derivation-building aren't implemented (substitutes, offloading,
etc). call-with-container is also in a semi-broken state, though the big
issues could be fixed by going back to the way it was and bind-mounting
a store directory to collect output in (bind-mounts mess with my head
for some reason... thanks for mentioning that, Ricardo). The list goes
on.

Maybe I should describe my experience a bit. Reading the C++ source was
extremely tedious. The only way to know what a value is at any point is
to follow the entire execution of the program up to that point. A
slightly faster heuristic is to rgrep for anything that might change
that value, and consider only those statements. This, however, is
confused by C++'s scoping rules. For example, there are 3 "settings"
variables visible in globals.cc: a global variable from globals.hh, a
private variable from globals.hh, and a namespace-local (I think? The
scoping rules still aren't completely clear to me) variable from
globals.cc. As a side-note, in the future I will consider any code that
repeats four of the same names in a row when read aloud ("Settings
settings; Settings::Settings()") to be something to run far away from.

The scheme side of things was nice. The most frustrating part was not
knowing what was already implemented, especially what was implemented
but used an RPC for a low-level task. I also didn't know that a scanner
already existed (in hindsight it should have been obvious - how else
would grafts work?) and so ended up writing my own inefficient version
(though I intuitively think for some reason that it could be made fast).

 I found myself checking the guile reference quite
frequently. One time I happened upon a part describing the ECMAScript
implementation (curiosity and all that), and noticed that several times
it was mentioned how irresponsible the implementor was. That scared me
probably more than it should have. By all objective criteria, I have
failed in this project, and it frightens me to think that, nice as the
community is, I could screw up badly enough to end up a footnote
somewhere as an example of what not to be. That fear pretty much sums up
my mental state during the second half of the summer.

In short, I've failed to achieve the goals I set out in my
proposal. Worst of all is that I failed to regularly communicate. I
specifically said that I would regularly communicate, and then I failed
to do so, especially during the second half of the summer. It seems that
the less I communicated, the more afraid of communicating I
became.

Currently nothing has been merged - I don't think it's in a fit state to
be merged presently. I intend to continue working on this as time
allows.  School started last week, though, and it's kept me quite busy
so far.

The code so far can be found at the guile-daemon branch on Savannah.

tl;dr - I did not achieve the goals I set out to achieve. That makes me
sad. I plan on working on it more.

- - reepca

-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAlmlGwUACgkQwWaqSV9/
GJybxwf/ZYtcANxxfaeS2YQFO3IQOlz+eI5cESvlzjyeQUZ6OgBm7sRTkSg67b2S
W4i3TvPKeEC/H1skFnYeh+ltlpQQo06ifgrh9ZUGNWbMeMTMq1Aqop6ZP97WQEtx
xihPCkjDve45zR4BwdqR6EB162c9Q16SJ3297g4SpcMKfoAKjnIv7T49tWCd2wCm
qp8Ia9D29DDYb8WAvD7ZyDo5g0fvB+qsHXpkNm5Lgic2m7XB0eRoWx/PzvtDNPzb
4z3OWN4vVvj1NZStyEgOkifuciY+Pn+W7RjcaQS0vrfkIcGFM7yZBm19nmBZAyrD
sz/vNcOfauVsMx3cI1MEZ7QAHIZfpg==
=GYqh
-END PGP SIGNATURE-

Re: Latest guile-daemon changes and bewilderment

[Caleb Ristvedt]

> Is there a line above or below the backtrace mentioning the uncaught
> exception?  Could you ‘strace -f’ the daemon process?

No, no line above or below. Very strange.

> BTW, I see the code uses ‘clone’ directly.  It would be safer to use
> ‘call-with-container’, which already handles bind mounts, non-local
> exits, and so on.  Would it be an option?

There are a couple of issues with using call-with-container. In
decreasing order of perceived difficulty to solve:

1. Copying the output from the build to the real store. How would that
work? It wouldn't work with call-with-container - the container can't
access the outside world, and the chroot directory is automatically
deleted once the scope of the container is left. On top of that, there's
no guarantee that anything inside the chroot directory is visible
outside of the container namespace, since it's on a separate filesystem
mounted in that separate namespace.

The primary solution I have in mind for this is to add a separate
procedure argument to call-with-container (maybe "use-output"?) to be
run after the main thunk has finished running but before the temporary
directory has been deleted and which takes the chroot directory as its
only argument. Also to change the mounting of a tmpfs on the chroot dir
to happen before the clone, and explicitly unmount it after running the
aforementioned use-output thunk.

2. Some MS_PRIVATE stuff. Cleaning up will fail when whatever filesystem
the chroot directory is on is mounted as MS_SHARED, which according to a
comment in build.cc is what systemd mounts stuff as. (I'm curious why we
don't seem to have run into this issue yet, perhaps I have misunderstood
something)

My solution here is to change the propagation type of the chroot
directory inside the namespace to MS_PRIVATE. Anything mounted under it
will inherit that propagation type and not appear mounted in the
original namespace, and unmounting the chroot directory should work
fine.

3. Miscellaneous order changes. I don't know enough about the inner
workings of linux to be completely sure to what extent the order of some
operations is significant.

4. Minor differences. For example, the C++ daemon doesn't currently
bind-mount /dev/ptmx or /dev/fuse or /dev/console. I don't think those
would be a problem, but I dunno.

... and then I paused writing this for 2 days while I checked whether my
in-theory solutions would work in practice. And it seems like they
actually do (see recent branch update). Mostly. I need to figure out why
it fails when a new user namespace is created - for some reason
pivot-root fails when new-root was mounted from a different user
namespace.

But on the bright side, it somehow solved the bug I described earlier. I
still haven't managed to make it all the way to building hello (a
libunistring test hangs), but it's getting much farther.


> Regarding scanning, (guix build grafts) contains a special-purpose
> reference scanner that Mark carefully optimized; it might be worth
> looking at.

Huh. I did not know that. In hindsight, it seems obvious that there must
have been something like that for grafts to function. I'll look into
that.

>> You'll notice that among the environment variables is
>> GUILE_AUTO_COMPILE=0. This is actually something I added myself because
>> for some reason the bootstrap guile wasn't honoring the
>> --no-auto-compile flag, but does honor the environment
>> variable. Strange.
>
> Yeah, we shouldn’t add this environment variable to derivations because
> that would influence everything that goes in there.

Aye, it was mostly just for debugging. That problem has also disappeared
with the switch to using call-with-container. It's nice and all, but I
do wish I knew what caused it.

> Could it be that ‘argv’ lacked the executable name as argv[0], and thus
> the argument list was shifted to the left?

That's what I thought too, but the same behavior happened when adding
the executable name explicitly as the first argument both to system* and
execl.

> Let’s maybe try to further debug this interactively on IRC.

... and then I promptly fell asleep and spent the next few days
(nights?) tinkering. Oops.

Oh well, there will be no shortage of debugging to be done. It seems
like that's going to be the pattern for the near future - add an
isolation mechanism or something that conforms better to what is
currently done, try to build stuff, get a bit farther, look for more
stuff to fix.

- reepca

Latest guile-daemon changes and bewilderment

[Caleb Ristvedt]

Hello guix!

After a pileup of changes and testing, I finally got around to pushing
my changes to the guile-daemon to the branch on savannah. The big
addition is half-working derivation-building (just one at a time, for
now, and only chroot builds for now) in the form of
guix/store/build-derivations.scm. More on the not-working half later.

This includes reference-scanning, which I think pretty much works, but
which I regret to say I spent too much time thinking about. Premature
optimization and all that. It takes a rather different approach to the
way the C++ code does it - it uses a trie to recognize references out of
a list of possibilities (namely, anything thrown in the build
environment's store). The idea is that additional pre-calculation can be
performed on each node and it can become a sort of branching skip table
of the sort used in that one string-matching algorithm whose name I
can't remember. At least, that's the way it works in my head - the parts
to make it fast haven't been implemented yet. I should probably split
the scanner out into a separate module.

The build environment currently is isolated by creating new pid, mount,
ipc, and uts namespaces, remounting everything as MS_PRIVATE, using
pivot-root to make a generated directory the new root, and unmounting
the old root.

So, does it work? Well... sort of. Which brings me to a really bizarre
problem to debug. *deep breath*

The way I've been going about testing this has been to first delete
test-tmp completely, then run (as root, since I'm testing chroot stuff):

./test-env guix build --dry-run hello
./test-env guile
scheme@(guile-user)> (use-modules (ice-9 readline))
scheme@(guile-user)> (activate-readline)
scheme@(guile-user)> ,m (guix store build-derivations)
scheme@(guile-user)> (build-derivation
  (read-derivation-from-file
   "path/to/guix/test-tmp/store/...hello-2.10.drv"))

This eventually fails trying to build the bootstrap make. This is the
only error message I get: http://paste.lisp.org/display/351519. So it
seems that some low-level guile thing is failing. I know from
print-statement debugging that the exact place it fails is when the
gnu-build-system calls apply to start the set-paths phase, but have no
indication of why this is. It doesn't make it to the set-paths
procedure, though. Perhaps someone more familiar with guile's internals
knows more?

Builder: http://paste.lisp.org/display/351521.
Environment variables: http://paste.lisp.org/display/351523.

You'll notice that among the environment variables is
GUILE_AUTO_COMPILE=0. This is actually something I added myself because
for some reason the bootstrap guile wasn't honoring the
--no-auto-compile flag, but does honor the environment
variable. Strange.

Stranger still, attempting to run the bootstrap guile interactively from
the build environment (replacing the execl with a system* so the build
environment can be explored from the REPL) causes it to print the
copyright notice and immediately exit with an exit code of 0. So
debugging that way isn't really an option. Notably, running the same
guile from outside of the build environment works fine.

Copy+pasting the builder code into the guile-2.2 REPL in the build
environment after adding the module-import to the load path works just
fine.

Confusing. I know there are several non-essential parts of the build
process that aren't implemented yet, but none of them are used by this
particular derivation as far as I can tell.

- reepca

GSoC project update 1

[Caleb Ristvedt]

Hello everyone. It's already almost July! Which means I've been working
on replacing the build daemon for about a month now. So it seems like a
good time for an update.

So far, I've replaced the register-path procedure in store.scm, which
means we now have some support for using sqlite (via guile-sqlite3 and
some utility macros) to access and modify the state database. We can
also now perform deduplication on individual store items. That's pretty
much all that's been produced so far code-wise (that you can see on the
guile-daemon branch at savannah, anyway), but that's okay, because I've
also gained a lot of knowledge.

Recently I've been trying to understand the details of how
nix/libstore/build.cc operates, and this has taught me the value of
trying to keep procedures no longer than one screen's height. It's also
taught me the value of immediately writing any insights I have down,
since the details, even of a single procedure I'm reading, are well past
the point where I can keep them all in my head.

Right now I'm working on implementing the building of derivations. There
are two main difficulties. The first is that the details of this process
(which files are included in the chroot by default, permissions,
ownership, environment variables, etc) are all in C++. I have discovered
that the picture I had of C++ going into this was, unfortunately, pretty
accurate. Here's a great example: reading through globals.cc, try to
figure out what "settings" refers to at each occurrence (I had a long
rant prepared, but that about sums it up).

The second, less obnoxious difficulty is that of code duplication. A lot
of the functionality of guix duplicates functionality from the daemon,
which was one of the motivations for the rewrite in the first place, but
the duplicated functionality is still built on top of remote procedure
calls, especially for accessing the database. For example,
topologically-sorted in store.scm is exactly what I wanted in order to
implement support for exportReferencesGraph, but it uses an RPC to get
the list of references of a store item. At the same time, there are
still many procedures that are what I need (derivations.scm is lovely),
so it's still worth checking if it's already implemented, but often a
miss.

My difficulties with Scheme are pretty minor, and the Guile reference
has been very useful. It took me awhile to realize that dynamic scoping
was very much not a thing in Scheme, as opposed to the sort-of-a-thing
state it's in in Common Lisp (I had a hack set up to shadow a hashing
function so that I could get the size of the thing being hashed as a
byproduct, but discovered it didn't work that way and ended up just
making a wrapper port that counted what went through it). I do get the
feeling when writing Scheme that set! is an unholy word, which makes for
a fun challenge, especially where iteration is involved. I want to
finally figure out how monads are supposed to be used, though. Perhaps I
should just read through the implementation in guix.

I think I have the build environment just about figured out (I've been
writing a detailed description of the process, which I plan to post once
it's finished), and can get to implementing derivation-building
soon. I'll start with assuming that a chroot is used, then extend it to
work with non-chroot builds later.

So yeah, that's the state of the project so far.

- reepca

Re: [PATCH] Prototype register-path

[Caleb Ristvedt]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I figured out why I was getting those two test failures after looking
more closely at tests/store.log: when I installed GuixSD, I did so
directly from another hard drive on the same system (having read in the
past about issues with multi-booting, I figured I'd just decide which to
boot to by selecting hard drives). I had previously cloned guix on the
Ubuntu hard drive, and figured I'd just mount it and copy it over. It
turns out that re-running make didn't alter the tests quite as much as I
thought it would, and the old path was stuck in some of them (on GuixSD
there is no code-downloads folder, it's just straight in
Programming/). After a git clean, though, the tests all pass.

> Though something went from with your email client.  :-)

Aye, turns out M-x epa-mail-sign doesn't work well with attachments. I
posted another message that should be signed correctly, as well as this
one.

> What are your thoughts on the next steps?

Well, I took a look at libstore/build.cc, and that is a lot of code. I
hope I can get away with only reading as much as is necessary to clarify
details. To that end, I'd like to make sure that my high-level
understanding of the build process is accurate.

- - A derivation is a node in a (acyclic?) graph of "what depends on what" or,
  alternately put, what is input to what.

- - Derivations can depend on other derivations or plain files (can those files 
be
  directories?), where "plain files" cannot depend on anything else.

- - "Building" a derivation involves first ensuring that all of its inputs are
  "built" (in the case of inputs that are derivations, this means ensuring its
  output exists as a store item, and if not, producing it by this same
  process. In the case of plain file inputs, this means just ensuring that they
  exist as a store item), and then preparing a build environment. The "builder"
  script is then executed in a chroot rooted under the build tree. After it has
  completed, the output (where is it put during the build process?) from
  building the derivation is placed in the store (and registered).

- - "preparing a build environment" involves first creating a directory in which
  the build will be run, then populating it with certain expected files and
  directories (as detailed in 2.4.1 of the manual). Also, the store items
  produced from building the inputs need to somehow be made available (is there
  a gnu/store/ under the build tree? I feel like I don't quite understand where
  everything goes in the build tree). Also, set any environment variables
  associated with the derivation.

Assuming I've got the gist of it sort of right, that leaves one more question:
how are hashes for the paths computed? It's not sha256 like the database hashes,
it seems to be 160 bits in size. How are all the inputs hashed together? Are all
the inputs put in one big directory and that is hashed in nar form, or what? And
what about store items that don't have any inputs (for example, something placed
in there with add-text-to-store)? I suppose in that case they are their own
input, perhaps?

Some of those answers might be in the manual, but I didn't manage to
find them.
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAllGRk0ACgkQwWaqSV9/
GJw9TQf/W8ewHEFzwTssGhMGW/pGsSvUMnzdQNmx31Y1HLMpUbhBazbIUa57pz5g
ZfDbUIRnKBYlCPi5X+8Om8fzMjWpUixYeyv/Clk3sRvbcQhod0cejbRIC0PAiTN0
mVlu4tUWJlEXekcG7OqhdZ8AZXJ53qOkNMEHR3h91PCJOZj2fOVwzlC5kp7gUfcM
C9a/rgCb6ZG6dXQNH3Q0qI38+akDg3tbD0BDk97yhHCBSVitwbJd9vTiEl3QhAPP
WFlZjmZGFpfruZiuRDyO8pOFC4tfdsEy/wv4spjq63ZgBNqLQsI9Y0JfXJWM/fEH
d+KsI54ix/yDhZnbZofLaogAH3Fo8w==
=Ukhf
-END PGP SIGNATURE-

Re: [PATCH] Prototype register-path

[Caleb Ristvedt]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hm, M-x epa-mail-verify says the signature is wrong. When I test it
without an attachment, it works fine. That's strange, I would have
thought epa-mail-sign would be aware of such things. Anyway, here's a
message that should verify correctly.
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAlk+KlAACgkQwWaqSV9/
GJwvIAgAmRbOYPMgLHOdQqhV7U8jHXdWvESP2Q6ZWMYSjo0XSMlgjgViEYFl/1CE
uLRtUKm6Lm2xUKznlUV1dJ3aZ6H35TtPxgjbkcpNaKUR9kMXIzZWnBkjhTHPKi6b
bNEOdnEUyQY4qokD5J1/X9HMQ9Nj2KqC6jU3uAtTZVVhW+o0fMrk8aysNL4oSHho
PHBXQMqOloze0c3Wa9tXnKK8JiMQ7xaEv7iGt2GZc65jqyjxW17ZACP0AOf77IhS
bHQvC6xY1ZNfbD5In+4l3M8OfZO3sdrDMYXRH0vIFDrFirdCD27KUuZ64WnLLu26
5P/JR0F5YCOXpZstKHUz/HbGim1zJw==
=H7zu
-END PGP SIGNATURE-

Re: [PATCH] Prototype register-path

[Caleb Ristvedt]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Apologies for the delay - in a shocking twist, it turns out professors
insisting on only one exit from functions had a point, and trying to
fully understand what optimisePath_, with its 14 exits and a goto, does,
was a challenge. I think I handle most of the main error cases (EEXIST,
ENOSPC, EMLINK) sort of okay now (the "shrug and ignore" sort of
exceptions are now caught, the rest are still propagated). I've also
created guix/store/deduplication.scm, and moved the relevant stuff
there. The tests pass to the same degree as before (2 failing, but
neither of them are register-path).

Here is my test-suite.log:



test-suite.log
Description: Binary data

When I create a new module that directly incorporates code written by
others (as in the case of moving reset-timestamps to (guix store
deduplication)), should I add anything to the copyright notice at the
top? Would that involve looking through the commit history to find
exactly who wrote that?

I read HACKING - does the part about waiting two weeks for feedback on
guix-patches before committing still apply to branches?

I've now updated my savannah account with a key, used to sign this.
-BEGIN PGP SIGNATURE-

iQEzBAEBCAAdFiEEdNapMPRLm4SepVYGwWaqSV9/GJwFAlk+InUACgkQwWaqSV9/
GJxpRAf+KWd4IQGdcGi4ret3GCfGJcfsmmZiSni8enzo4EdKmxT4MOf/FYwnrOHF
ZB33YSc2mlhxPKs1y2soWHsPYyhODVL2sy3jQ4CKSpINZNqb0iHts8zfDSDRQoKi
krOzPZTB7o1eOHYDS9xXnwqoNdGRzwG1KrdQZYCAIg6/UdkFaWevSn+Vo1LgzRp+
IFUxrLj92JxBraiyhQV4Uf7hALhA3WWKOym/9K5WN/NO8x4Somv4nIdrMFS21euq
qQbPzZ+5gb0m8k0smH6JC3E7DkPXL0UYTdbdsr4aGxIIFvwy6qLmDyBKZR7ieDjM
x2fTJp20v+28BV0Gzbtk4LnK1v59zA==
=1ttQ
-END PGP SIGNATURE-

Re: [PATCH] Prototype register-path

[Caleb Ristvedt]

l...@gnu.org (Ludovic Courtès) writes:

> Hi reepca!
>
> I gave this patch set a try and looked at the code, and it looks very
> good to me!
>
> “make check TESTS=tests/store.scm” passes now with the fixes you posted
> today (‘getenv’ & co.)

I end up with 2 failed tests, but neither of them are the register-path
one. 

> For deduplication, you already know the (guix hash) API that will allow
> you to do that I guess.  The “deduplication database” is simply the
> /gnu/store/.links directory.  Each entry there is a hard link to a file;
> the name of the entry is the hex representation of the sha256 hash of
> that file.  So when inserting a file in the store, all you need is to
> look up its hash in /gnu/store/.links and hard-link from there.  See
> what I mean?

And if it isn't in /gnu/store/.links, put a hardlink there? Also, when
you say "when inserting a file", what do you mean by that? I was under
the impression that the paths given to register-path were already in the
store but simply weren't in the database or otherwise "officially" in
the store. Is my assumption incorrect?

Less on-topic, I don't really know how sha256 works, but if it is a
hashing function, there must be a possibility (albeit small) of
collisions, right? Is it worth developing a strategy for handling them?

> Some minor suggestions about the code:

Implemented all of these except maybe half of the commit log one... I'm
not sure what the commit message headline should be, so I sort of
guessed based on some of the ones in recent history, and I'm not sure
I've got the formatting quite right yet, but it should be a lot closer.


> What about pushing your changes to a WIP branch on Savannah or
> elsewhere?  (If you have an account on Savannah we can give you access.)

I just made an account today as "reepca". Should I submit a "request for
inclusion"? In the meantime, here's a much less fixup-riddled patch set
that I think is just about feature-complete as far as register-path is
concerned:

>From ce4a322446d1865791686b1e4573973573bdcdfc Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Sat, 3 Jun 2017 02:26:05 -0500
Subject: [PATCH 1/7] guix: register-path: Implement prototype in scheme.

* guix/store.scm (register-path): reimplement in scheme.
* guix/sql.scm: New file.
---
 gnu/packages/package-management.scm |   3 +-
 guix/sql.scm| 224 
 guix/store.scm  |  78 ++---
 3 files changed, 286 insertions(+), 19 deletions(-)
 create mode 100644 guix/sql.scm

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index af91ec1d7..50be3a23f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -250,7 +250,8 @@
   (propagated-inputs
`(("gnutls" ,gnutls/guile-2.2) ;for 'guix download' & co.
  ("guile-json" ,guile-json)
- ("guile-ssh" ,guile-ssh)))
+ ("guile-ssh" ,guile-ssh)
+ ("guile-sqlite3" ,guile-sqlite3)))
 
   (home-page "https://www.gnu.org/software/guix/";)
   (synopsis "Functional package manager for installed software packages and versions")
diff --git a/guix/sql.scm b/guix/sql.scm
new file mode 100644
index 0..b1e0c0aa4
--- /dev/null
+++ b/guix/sql.scm
@@ -0,0 +1,224 @@
+(define-module (guix sql)
+  #:use-module (sqlite3)
+  #:use-module (system foreign)
+  #:use-module (rnrs bytevectors)
+  #:use-module (srfi srfi-9)
+  #:export (sqlite-register))
+
+;; Miscellaneous SQL stuff, currently just setup for sqlite-register. Mostly
+;; macros.
+
+;; This really belongs in guile-sqlite3, as can be seen from the @@s.
+(define sqlite-last-insert-rowid
+  (let ((last-rowid (pointer->procedure
+ int
+ (dynamic-func "sqlite3_last_insert_rowid"
+   (@@ (sqlite3) libsqlite3))
+ (list '*
+(lambda (db)
+  "Gives the row id of the last inserted row in DB."
+  (last-rowid ((@@ (sqlite3) db-pointer) db)
+
+
+;; Should I go from key->index here or try to change that in guile-sqlite3?
+(define-syntax sql-parameters
+  (syntax-rules ()
+"Converts key-value pairs into sqlite bindings for a specific statement."
+((sql-parameters statement (name1 val1) (name2 val2) (name3 val3) ...)
+ (begin (sqlite-bind statement name1 val1)
+(sql-parameters statement (name2 val2) (name3 val3) ...)))
+((sql-parameters statement (name value))
+ (sqlite-bind statement name value
+
+(define* (step-all statement #:optional (callback noop))
+  "Step until statement is completed. Return number of rows."
+  ;; Where "number of rows" 

Re: [PATCH] Prototype register-path

[Caleb Ristvedt]

I think I may have accidentally left some stuff out of that patch due to
being new to "git format-patch" - I did several fixups which I'm not
sure were included. Anyway, I've changed it so that register-path now
honors the NIX_* environment variables that are relevant to it, which
means it now passes "make check TESTS=tests/store.scm". Still quite a
bit more to do, mainly resetting timestamps/permissions and
deduplication.

I'm especially pleased with how the interaction between environment
variables, parameters, and defaults is now a straightforward priority
list using a cond. Said interactions were spread out across the C++
codebase using global variables (well, one global variable) and lots of
state-changing, so I feel pretty good about getting that in one place.

A question about protocol, though - should followup patches like this be
replies or new top-level posts? And how often should I send them?

Again, comments welcome.

>From 90f250814b1456387b8b0341b1f245a1c4e05f66 Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Sat, 3 Jun 2017 02:26:05 -0500
Subject: [PATCH 1/7] Implement prototype register-path in scheme

New file and module sql.scm: mostly utility macros for sqlite-register.
---
 gnu/packages/package-management.scm |   3 +-
 guix/sql.scm| 219 
 guix/store.scm  |  77 ++---
 3 files changed, 280 insertions(+), 19 deletions(-)
 create mode 100644 guix/sql.scm

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index af91ec1d7..50be3a23f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -250,7 +250,8 @@
   (propagated-inputs
`(("gnutls" ,gnutls/guile-2.2) ;for 'guix download' & co.
  ("guile-json" ,guile-json)
- ("guile-ssh" ,guile-ssh)))
+ ("guile-ssh" ,guile-ssh)
+ ("guile-sqlite3" ,guile-sqlite3)))
 
   (home-page "https://www.gnu.org/software/guix/";)
   (synopsis "Functional package manager for installed software packages and versions")
diff --git a/guix/sql.scm b/guix/sql.scm
new file mode 100644
index 0..ae4a1d27d
--- /dev/null
+++ b/guix/sql.scm
@@ -0,0 +1,219 @@
+(define-module (guix sql)
+  #:use-module (sqlite3)
+  #:use-module (system foreign)
+  #:use-module (rnrs bytevectors)
+  #:use-module (srfi srfi-9)
+  #:export (sqlite-register))
+
+;; Miscellaneous SQL stuff, currently just setup for sqlite-register. Mostly
+;; macros.
+
+;; This really belongs in guile-sqlite3, as can be seen from the @@s.
+(define sqlite-last-insert-rowid
+  "Gives the row id of the last inserted row in DB."
+  (let ((last-rowid (pointer->procedure
+ int
+ (dynamic-func "sqlite3_last_insert_rowid"
+   (@@ (sqlite3) libsqlite3))
+ (list '*
+(lambda (db)
+  (last-rowid ((@@ (sqlite3) db-pointer) db)
+
+
+;; Should I go from key->index here or try to change that in guile-sqlite3?
+(define-syntax sql-parameters
+  "Converts key-value pairs into sqlite bindings for a specific statement."
+  (syntax-rules ()
+((sql-parameters statement (name1 val1) (name2 val2) (name3 val3) ...)
+ (begin (sqlite-bind statement name1 val1)
+(sql-parameters statement (name2 val2) (name3 val3) ...)))
+((sql-parameters statement (name value))
+ (sqlite-bind statement name value
+
+(define* (step-all statement #:optional (callback noop))
+  "Step until statement is completed. Return number of rows."
+  ;; Where "number of rows" is assumed to be number of steps taken, excluding
+  ;; the last one.
+  (let maybe-step ((ret (sqlite-step statement))
+   (count 0))
+(if ret
+(maybe-step ret (+ count 1))
+count)))
+
+;; I get the feeling schemers have probably already got this "with" business
+;; much more automated than this...
+(define-syntax with-sql-statement
+  "Automatically prepares statements and then finalizes statements once the
+scope of this macro is left. Also with built-in sqlite parameter binding via
+key-value pairs."
+  (syntax-rules ()
+((with-sql-statement db sql statement-var
+ ((name1 val1) (name2 val2) ...)
+ exps ...)
+ (let ((statement-var (sqlite-prepare db sql)))
+   (dynamic-wind noop
+ (lambda ()
+   (sql-parameters statement-var
+(name1 val1)
+(name2 val2) ...)
+   exps ...)
+ (lambda ()
+   (sqlite-finalize statement-var

[PATCH] Prototype register-path

[Caleb Ristvedt]

So far I've got the main functionality of register-path working: it
successfully puts the passed information in a database (at least in my
manual tests it did), and it even puts it in the right database based on
prefix and state-directory. But there are quite a few side-effects
missing, most of them noted above register-path in store.scm. There's
deduplication, resetting timestamps and permissions and stuff (the C++
code called it "canonicalizing"), actually returning true in the case of
success like the documentation says it should, processing environment
variables, and figuring out if I should be automatically creating the
database if it doesn't exist yet.

Currently the register-path unit test fails due to being unable to open
the database for writing - the test doesn't specify a prefix or
state-directory, so it's trying to open /var/guix/db/db.sqlite, which
requires root access to open for writing. Perhaps the test is using an
environment variable or similar to specify a different database, which I
haven't implemented yet.

Comments welcome.

>From f4af8973a7b41664130b05bbe8a4069f62a087c3 Mon Sep 17 00:00:00 2001
From: Caleb Ristvedt 
Date: Sat, 3 Jun 2017 02:26:05 -0500
Subject: [PATCH] Implement prototype register-path in scheme

New file and module sql.scm: mostly utility macros for sqlite-register.
---
 gnu/packages/package-management.scm |   3 +-
 guix/sql.scm| 219 
 guix/store.scm  |  77 ++---
 3 files changed, 280 insertions(+), 19 deletions(-)
 create mode 100644 guix/sql.scm

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index ceaf51b67..80ffe0a2e 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -249,7 +249,8 @@
   (propagated-inputs
`(("gnutls" ,gnutls/guile-2.2) ;for 'guix download' & co.
  ("guile-json" ,guile-json)
- ("guile-ssh" ,guile-ssh)))
+ ("guile-ssh" ,guile-ssh)
+ ("guile-sqlite3" ,guile-sqlite3)))
 
   (home-page "https://www.gnu.org/software/guix/";)
   (synopsis "Functional package manager for installed software packages and versions")
diff --git a/guix/sql.scm b/guix/sql.scm
new file mode 100644
index 0..ae4a1d27d
--- /dev/null
+++ b/guix/sql.scm
@@ -0,0 +1,219 @@
+(define-module (guix sql)
+  #:use-module (sqlite3)
+  #:use-module (system foreign)
+  #:use-module (rnrs bytevectors)
+  #:use-module (srfi srfi-9)
+  #:export (sqlite-register))
+
+;; Miscellaneous SQL stuff, currently just setup for sqlite-register. Mostly
+;; macros.
+
+;; This really belongs in guile-sqlite3, as can be seen from the @@s.
+(define sqlite-last-insert-rowid
+  "Gives the row id of the last inserted row in DB."
+  (let ((last-rowid (pointer->procedure
+ int
+ (dynamic-func "sqlite3_last_insert_rowid"
+   (@@ (sqlite3) libsqlite3))
+ (list '*
+(lambda (db)
+  (last-rowid ((@@ (sqlite3) db-pointer) db)
+
+
+;; Should I go from key->index here or try to change that in guile-sqlite3?
+(define-syntax sql-parameters
+  "Converts key-value pairs into sqlite bindings for a specific statement."
+  (syntax-rules ()
+((sql-parameters statement (name1 val1) (name2 val2) (name3 val3) ...)
+ (begin (sqlite-bind statement name1 val1)
+(sql-parameters statement (name2 val2) (name3 val3) ...)))
+((sql-parameters statement (name value))
+ (sqlite-bind statement name value
+
+(define* (step-all statement #:optional (callback noop))
+  "Step until statement is completed. Return number of rows."
+  ;; Where "number of rows" is assumed to be number of steps taken, excluding
+  ;; the last one.
+  (let maybe-step ((ret (sqlite-step statement))
+   (count 0))
+(if ret
+(maybe-step ret (+ count 1))
+count)))
+
+;; I get the feeling schemers have probably already got this "with" business
+;; much more automated than this...
+(define-syntax with-sql-statement
+  "Automatically prepares statements and then finalizes statements once the
+scope of this macro is left. Also with built-in sqlite parameter binding via
+key-value pairs."
+  (syntax-rules ()
+((with-sql-statement db sql statement-var
+ ((name1 val1) (name2 val2) ...)
+ exps ...)
+ (let ((statement-var (sqlite-prepare db sql)))
+   (dynamic-wind noop
+ (lambda ()
+   (sql-parameters statement-var
+(name1 val1)
+(name2 val2) ...)
+   exps

Re: reepca joins Guix for GSoC 2017!

[Caleb Ristvedt]

Hello everyone. I'm the aforementioned reepca (pronounced reep-kay). Nice
to meet you all, and I look forward to working on this project!

On Fri, May 5, 2017 at 3:50 PM, Ludovic Courtès  wrote:

> Hello Guix!
>
> I’m happy to announce that reepca (Cc’d) will join us for the Google
> Summer of Code (GSoC).  Reepca will be working on rewriting the build
> daemon in Scheme:
>
>   https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00059.html
>   https://libreplanet.org/wiki/Group:Guix/GSoC-2017#Rewrite_
> build_daemon_in_Guile_Scheme
>
> Formally Ricardo and I will be mentoring, but of course everyone on this
> list and on IRC is welcome to chime in and provide guidance and
> feedback.
>
> Reepca: do not hesitate to ask any questions you may have on #guix or
> help-g...@gnu.org or guix-devel@gnu.org.  We’re here to help!
>
> Congratulations and welcome to Guix, reepca!  :-)
>
> Ludo’.
>

GSoC proposal last-minute double-check

[Caleb Ristvedt]

I realize this is very last-minute indeed, but thought I'd see if I could
get some feedback before I submit. I'm looking at working on the Guile
Scheme guix-daemon - it seems much more concrete in scope than the build
tool. I wrote my proposal based on
https://www.gnu.org/software/soc-projects/guidelines.html, but I think that
might be a bit outdated - it's still got links to 2013 stuff and talks
about mid-semester evaluations even though the schedule currently is to
divide the summer in to 3 work periods. I'm least confident about the
timeline of my proposal - is the time for those goals reasonable, evenly
spaced, etc?

Here's the full proposal as I have it now:

* Name
  Caleb Ristvedt (irc nickname: reepca)

* email address
  caleb.ristv...@cune.org

* name of project
  Guix Rewrite build daemon in Guile Scheme

* Summary
  Currently the build daemon of Guix is written in C++, inherited from Nix.
It
  works fine but is not as hackable as we'd like, and has poor integration
with
  the rest of Guix. For instance, the daemon calls out to the 'guix
substitute',
  'guix authenticate', and 'guix offload' commands, but its interface to
these
  commands is very limited. Furthermore, a large part of the daemon's code
is
  already implemented in Scheme: container functionality is available with
  'call-with-container', archive creation is implemented in (guix nar),
writing
  derivation files (.drv) is done in (guix derivations), and so on.

  The goal of this project would be to rewrite the daemon in Guile Scheme
using
  the building blocks already available. Important missing bits include the
  garbage collector and its scanner (which scans files for references to
  /gnu/store items), the scheduler, which schedules derivation builds in
  topological order, using the specified number of cores, etc.

* Benefits
  This would benefit users indirectly by benefiting developers. The easier
it is
  to understand and modify the code, the better the quality of the resulting
  software (I leave this as an axiom). Better-quality software benefits
  users. So this project will benefit users.
** Better hackability
   By "hackability", I mean ease of modifying. Where more than one language
   needs to be known to make changes, the code is more difficult to
   change. Where one of those languages is C++, it's even more difficult to
   change. Where multiple languages need to talk to each other, changing the
   interface is more difficult than if the interface involves only one
   language.
** Reduce duplication of code
   Duplicated code is more code than if there is not duplication. It's
easier to
   understand less code. If we can replace the C++ implementation with one
in
   Guile Scheme, there won't be duplication (we can get rid of the C++
   implementation). So replacing the C++ implementation with one in Guile
Scheme
   will make the code easier to understand.

* Deliverables
  A program which, when run, will perform the same function as the current
  guix-daemon, and is written by me solely in Guile Scheme. Additionally,
  documentation explaining how the garbage collector and scheduler work and
  analyzing the time and space requirements of the algorithms used as well
as
  profiling the real-world performance of the daemon.

* Plan
  (I can't find any mention of mid-term evaluations in the 2017 GSoC
timeline,
  so I'm guessing that
https://www.gnu.org/software/soc-projects/guidelines.html
  is outdated?)

  The missing pieces according to the suggestion description include the
garbage
  collector, scanner, and scheduler. Presumably those are either the only
  missing pieces or the only non-trivial missing pieces.

  I will read through the current C++ code implementing garbage collection
and
  scanning during the "Community Bonding Period" to learn how it works.

  The first step should be to implement the missing pieces. I will be on
  schedule if I have the garbage collector and scanner working ("working"
here
  means that given sufficient time, all packages lacking references will be
  deleted or otherwise noticed by a program I wrote in Guile Scheme) by the
end
  of the first work period, and if the scheduler is working by the end of
the
  second work period ("working" here means that a program I wrote in Guile
  Scheme can order derivations topologically and schedule their building
with
  the same characteristics as the current guix-daemon can). By the end of
the
  third work period I will be on schedule if a complete executable,
  functioning in the same way as the current guix-daemon and written by me
in
  Guile Scheme, exists.

  The results from the first two work periods should be usable by
themselves in
  the event that I am unable to complete the entire project.

** Other commitments
   During the summer I will be unable to work from 1300 to 1800 UTC on
Sundays

GSoC build tool clarification

[Caleb Ristvedt]

The description of "replaces the likes of autools, cmake" makes sense, but
I'm not sure I understand the more advanced features being described -
specifically "... should ultimately be able to run complex workflows on
multiple servers...". Does that just mean that the build process should be
able to be distributed across systems, or is there more to that implied by
the "complex" part?

By the way, hi everyone!