Re: Finding the store path of a package

[Nikolaos Chatzikonstantinou]

>Ludovic Courtès wrote:
>> Yes.  In the presence of grafts, run “guix build PKG”.  That always
>> gives you the store file name of PKG, 100% reliable!
>At the cost of a few hours of CPU time, in the worst case.
>> I regularly do things like:
>>
>>   ls $(guix build PKG)/bin
>>   find $(guix build PKG) -name …
>What I am looking for is the equivalent of
>   ls $(guix build PKG)
>that fails in whatever way for packages that are not in the store, but
>guarantees (1) not adding anything to the store and (2) response times
>short enough for interactive user interfaces.

You can also try guix size $PKG.

Regards,
Nikolaos Chatzikonstantinou

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Why #:graft? #f?  Because if you enable graft, you’ll potentially have
> to build/download the thing, and that wouldn’t buy you anything because
> the set of file names is the same in the grafted package.

Four weeks later: this mostly works, but sometimes fails (by
downloading/building package) and it rarely fails completely
(error message). And I have no idea what is going on.

As an experiment, I ran the attached script via "guix repl". It requests
and displays the store paths for all non-hidden packages. Using
"guix gc" before and after, I found that it downloads/builds
more than 5 store entries with a total size of 7.5 GB.

One small but interesting example: ABCL

##
Store paths for abcl@1.8.0:
fetching path `/gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2...
 python-3.8.2  12.0MiB1.8MiB/s 00:07 [##] 100.0%

fetching path 
`/gnu/store/bvd09gb8ka642jzgxd2lpqlpdp160gn0-python-wrapper-3.8.2'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/bvd09gb8ka642jzgxd2lpqlpdp160gn0-python-wrapper-3.8.2...
 python-wrapper-3.8.2  347B   177KiB/s 00:00 [##] 100.0%

fetching path `/gnu/store/h8z924ip7ialjhd3sc2id5yh0jy7cj20-python-nose-1.3.7'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/h8z924ip7ialjhd3sc2id5yh0jy7cj20-python-nose-1.3.7...
 python-nose-1.3.7  197KiB1.4MiB/s 00:00 [##] 100.0%

fetching path `/gnu/store/cshy0265w5ifh12v9dcmlrdd3cwb61db-mercurial-5.6.1'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/cshy0265w5ifh12v9dcmlrdd3cwb61db-mercurial-5.6.1...
 mercurial-5.6.1  3.3MiB  3.1MiB/s 00:01 [##] 100.0%

((out . "/gnu/store/nz8xi3x7n3v2k4s4piiw7r8plvxam9qz-abcl-1.8.0"))
##

Why does computing the store path for a Java package require fetching
Mercurial (and thus Python)? The only link I see is icedtea-6, whose
build procedure apparently uses Mercurial to download the OpenJDK
sources. Probably this gets inherited by icedtea-8, which is used for
building ABCL. But... why does it take Mercurial to just compute the
store path??? If my understanding of Guix hashes is correct (which I am
beginning to doubt), it shouldn't require more than the hash of the
Mercurial package.

As an example for a hard failure, see sunxi-tools@1.4.2. Asking for
its store paths downloads plenty of stuff, before failing with:

  while setting up the build environment: a `armhf-linux' is required to build 
`/gnu/store/4blcfrrzc3yx6xjvw31mhnr1r9lpw4mj-Python-3.8.2.tar.xz.drv', but I am 
a `x86_64-linux'

That makes sense, of course, and I don't mind getting an error in such a
case, but I'd prefer it to fail without first downloading tons of stuff,
including the heavyweight gcc-7.5.0.

Cheers,
  Konrad.



display-store-paths.scm
Description: Binary data

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Here’s an example of how to do that:

Works fine, thanks!

> Why #:graft? #f?  Because if you enable graft, you’ll potentially have
> to build/download the thing, and that wouldn’t buy you anything because
> the set of file names is the same in the grafted package.

OK, so that's the secret, because that's the only difference with what
I tried before.

One day I'll figure out how grafts work, but that day is not today ;-)

> For #2, there have been discussions about building a service that would
> create such a database—a mapping from file names to packages.  It’s not
> possible to do with purely local knowledge because, by definition, you’d
> have to build/download every package.  I don’t think it has materialized
> yet, though.

That would certainly be the best solution, but in the meantime, I'll go
ahead with what is possible today. In practice, I expect most such
queries to succeed because there are only a few packages that contain
popular commands, and those are usually in the store.

Cheers,
  Konrad.

Re: Finding the store path of a package

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen  skribis:

>> Yes.  In the presence of grafts, run “guix build PKG”.  That always
>> gives you the store file name of PKG, 100% reliable!
>
> At the cost of a few hours of CPU time, in the worst case.
>
>> I regularly do things like:
>>
>>   ls $(guix build PKG)/bin
>>   find $(guix build PKG) -name …
>
> What I am looking for is the equivalent of
>
>ls $(guix build PKG)
>
> that fails in whatever way for packages that are not in the store, but
> guarantees (1) not adding anything to the store and (2) response times
> short enough for interactive user interfaces.
>
>> If you want a variant that does that without building/downloading it,
>> it’s also possible, though not as easily from the command line.
>
> Guile is fine, no problem. But so far, I haven't found anything even at
> the Guile level that respects my two conditions.

Here’s an example of how to do that:

--8<---cut here---start->8---
scheme@(guix-user)> ,use(guix)
scheme@(guix-user)> (define s (open-connection ))
scheme@(guix-user)> ,use(gnu packages base)
scheme@(guix-user)> (package-derivation s coreutils #:graft? #f)
$1 = # 
/gnu/store/yvsd53rkbvy9q8ak6681hai62nm6rf31-coreutils-8.32-debug 
/gnu/store/n8awazyldv9hbzb7pjcw76hiifmvrpvd-coreutils-8.32 7fc814f2e1e0>
scheme@(guix-user)> (derivation-outputs $1)
$2 = (("debug" . #< path: 
"/gnu/store/yvsd53rkbvy9q8ak6681hai62nm6rf31-coreutils-8.32-debug" hash-algo: 
#f hash: #f recursive?: #f>) ("out" . #< path: 
"/gnu/store/n8awazyldv9hbzb7pjcw76hiifmvrpvd-coreutils-8.32" hash-algo: #f 
hash: #f recursive?: #f>))
scheme@(guix-user)> (derivation->output-path $1 "out")
$3 = "/gnu/store/n8awazyldv9hbzb7pjcw76hiifmvrpvd-coreutils-8.32"
--8<---cut here---end--->8---

Why #:graft? #f?  Because if you enable graft, you’ll potentially have
to build/download the thing, and that wouldn’t buy you anything because
the set of file names is the same in the grafted package.

Does that make sense?

> Background: I am working on a interactive UI for running reproducible
> computations via Guix:
>
>   https://github.com/khinsen/guix-gtoolkit/

Nice!

> I'd like to implement (1) browsing package contents ("what exactly do I
> get by adding "core-utils" to my environment?") and (2) searching
> packages by the files they contain ("which package do I have to add to
> my environment to get the ls command?"). There will be a button for
> explicitly building a package, but I don't want it to happen as a side
> effect when doing operations that need to be fast.

For #2, there have been discussions about building a service that would
create such a database—a mapping from file names to packages.  It’s not
possible to do with purely local knowledge because, by definition, you’d
have to build/download every package.  I don’t think it has materialized
yet, though.

Thanks,
Ludo’.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

> On a side note, Ricardo did recently some stuff as UI for packages,
>
>   
>   

Looks good! My project has a lot of overlap, except that it is very
intentionally not based on Web technology, but on a malleable platform
(http://gtoolkit.com/) in which you can also do data analysis and other
computation. It's more like Emacs than like a Web app.

> And on another side note, I would like to have the abilities to join or
> intersect graphs; be able to visualize “guix graph foo bar” using D3.js,
> or the intersection or the complementary of the intersection, etc.
> Something similar for packages as the Ludo’s proof of concept for
> services,
>
>

Nice! That kind of visualization is not my focus for now, but the people
who build the Glamorous Toolkit platform are very much into that and
there's a lot of support code for visually exploring software systems:

   https://gtoolkit.com/usecases/software-assessment/

Cheers,
  Konrad.

Re: Finding the store path of a package

[zimoun]

Hi Konrad,

On Mon, 22 Mar 2021 at 08:39, Konrad Hinsen  wrote:

> Background: I am working on a interactive UI for running reproducible
> computations via Guix:
>
>   https://github.com/khinsen/guix-gtoolkit/
>
> I'd like to implement (1) browsing package contents ("what exactly do I
> get by adding "core-utils" to my environment?") and (2) searching
> packages by the files they contain ("which package do I have to add to
> my environment to get the ls command?"). There will be a button for
> explicitly building a package, but I don't want it to happen as a side
> effect when doing operations that need to be fast.

Nice!

On a side note, Ricardo did recently some stuff as UI for packages,

  
  

And on another side note, I would like to have the abilities to join or
intersect graphs; be able to visualize “guix graph foo bar” using D3.js,
or the intersection or the complementary of the intersection, etc.
Something similar for packages as the Ludo’s proof of concept for
services,

   

Sadly, time is not respecting what we are doing without considering
him. ;-)

Cheers,
simon

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Ludo,

> Yes.  In the presence of grafts, run “guix build PKG”.  That always
> gives you the store file name of PKG, 100% reliable!

At the cost of a few hours of CPU time, in the worst case.

> I regularly do things like:
>
>   ls $(guix build PKG)/bin
>   find $(guix build PKG) -name …

What I am looking for is the equivalent of

   ls $(guix build PKG)

that fails in whatever way for packages that are not in the store, but
guarantees (1) not adding anything to the store and (2) response times
short enough for interactive user interfaces.

> If you want a variant that does that without building/downloading it,
> it’s also possible, though not as easily from the command line.

Guile is fine, no problem. But so far, I haven't found anything even at
the Guile level that respects my two conditions.

Background: I am working on a interactive UI for running reproducible
computations via Guix:

  https://github.com/khinsen/guix-gtoolkit/

I'd like to implement (1) browsing package contents ("what exactly do I
get by adding "core-utils" to my environment?") and (2) searching
packages by the files they contain ("which package do I have to add to
my environment to get the ls command?"). There will be a button for
explicitly building a package, but I don't want it to happen as a side
effect when doing operations that need to be fast.

Cheers,
  Konrad.

Re: Finding the store path of a package

[Ludovic Courtès]

Hi!

Konrad Hinsen  skribis:

>>> does “guix build  -n” fit your use-case?
>>
>> Checking... yes! I hadn't even considered using "build" when building is
>> exactly what I do not want to happen. But yes, it works just fine.
>
> Not quite:
>
> $ guix build -n zziplib
> substitute: 
> /gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
> warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
> 24,6 MB would be downloaded:
>/gnu/store/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i
>/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
>/gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
>/gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2
>/gnu/store/7hizrpdsqf6q3pjgzmi51r5vbzlijkw0-python-minimal-wrapper-3.8.2
>
> My first impression is that "guix build -n" shows the store path of the
> package only if all dependencies are in the store. Otherwise, it shows
> the dependencies.

Yes.  In the presence of grafts, run “guix build PKG”.  That always
gives you the store file name of PKG, 100% reliable!

I regularly do things like:

  ls $(guix build PKG)/bin
  find $(guix build PKG) -name …

But note that ‘guix build’ returns one file name per output, so in case
of multiple-output packages, you have to pipe its output through ‘head’,
‘grep’, or similar.

If you want a variant that does that without building/downloading it,
it’s also possible, though not as easily from the command line.

Ludo’.

Re: Finding the store path of a package

[zimoun]

Hi,

In addition, I am also lost with this sequence:

 1. dry-run openmpi list dependencies
 2. do the wrong thing with the REPL
 3. garbage collect
 4. redo the dry-run and now the dependencies are not listed
 
Sorry Konrad, ’package-output’ is definitively wrong.  Worse, the items
listed in the first dry-run and then downloaded are not in the store but
not listed by dry-run neither.

--8<---cut here---start->8---
$ guix build -n openmpi
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
16.0 MB would be downloaded:


   /gnu/store/gm6d1vr75d4247bq834pks4ln8j3bm3l-psm2-11.2.86
   /gnu/store/4i9xg8dwns6ksjcb2kxkxmj9100jl4x5-ucx-1.6.1
   /gnu/store/977xygcarq06p8qsxkj686cxchf6m6pg-opensm-3.3.22
   /gnu/store/hbamhqhm4vq8fp27v9qgyc9rr6pf7fyw-libnl-3.5.0
   /gnu/store/1sw0s2a1ndqzvngmf1jsn19nwk96m9f9-rdma-core-33.1
   /gnu/store/wlhizxq9xcfz7vdf5h75xcalabad81q6-libfabric-1.11.2
   /gnu/store/y88fpc6jk6a4smxqjq0s835q5mximf02-linux-pam-1.5.1
   /gnu/store/agmdf3ib9sx7wqpznxxyki0pdbyhi4p5-freeipmi-1.6.7
   /gnu/store/b0x9j72z490kk0pld2ab4qa146gq4x00-numactl-2.0.14
   /gnu/store/jfwfsq932r209myi2ybrhy1yzh86m6a7-json-c-0.14
   /gnu/store/1s9gmhycv7xhzgym0w13rnbyv4irk2d7-hwloc-2.4.1-lib
   /gnu/store/vnyxs3vvzday63hk0qmlxfaq0wikc11h-munge-0.5.14
   /gnu/store/4lnskfwx6s7k8d3gr3mdq1p27z784m1r-slurm-20.11.3
   /gnu/store/yzs7m97m0d224qf1x8qxg1y3j5n29c3b-psm-3.3.20170428
   /gnu/store/wqcbimmq44rk882hmkj99zh43jmamps2-openmpi-4.0.5-debug
   /gnu/store/g07826knh72xwsdvpim5k30pqzjkcry9-openmpi-4.0.5
--8<---cut here---end--->8---

--8<---cut here---start->8---
$ guix repl 
  
GNU Guile 3.0.5 

Copyright (C) 1995-2021 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help. 


scheme@(guix-user)> ,use(gnu packages mpi)
scheme@(guix-user)> ,use(guix store)
scheme@(guix-user)> ,use(guix packages)
scheme@(guix-user)> (with-store store (package-output store openmpi))
fetching path `/gnu/store/1s9gmhycv7xhzgym0w13rnbyv4irk2d7-hwloc-2.4.1-lib'...
Downloading 
https://ci.guix.gnu.org/nar/lzip/1s9gmhycv7xhzgym0w13rnbyv4irk2d7-hwloc-2.4.1-lib...
hwloc-2.4.1-lib  204KiB  513KiB/s 00:00 [##]100.0%
[..]
$1 = "/gnu/store/g07826knh72xwsdvpim5k30pqzjkcry9-openmpi-4.0.5"
scheme@(guix-user)> ,q
--8<---cut here---end--->8---

--8<---cut here---start->8---
$ guix gc
--8<---cut here---end--->8---

--8<---cut here---start->8---
$ guix build -n openmpi
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
12.5 MB would be downloaded:
   /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2
   /gnu/store/7hizrpdsqf6q3pjgzmi51r5vbzlijkw0-python-minimal-wrapper-3.8.2
--8<---cut here---end--->8---


--8<---cut here---start->8---
$ guix show openmpi | recsel -p dependencies | grep slurm
+ rdma-core@33.1 slurm@20.11.3 ucx@1.6.1 valgrind@3.16.1

$ guix gc -D /gnu/store/4lnskfwx6s7k8d3gr3mdq1p27z784m1r-slurm-20.11.3
finding garbage collector roots...
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 13699.88 MiB

$ guix gc --list-dead | grep slurm
finding garbage collector roots...
determining live/dead paths...

$ guix gc --list-live | grep slurm
finding garbage collector roots...
determining live/dead paths...

$ ls /gnu/store/4lnskfwx6s7k8d3gr3mdq1p27z784m1r-slurm-20.11.3
ls: cannot access '/gnu/store/4lnskfwx6s7k8d3gr3mdq1p27z784m1r-slurm-20.11.3': 
No such file or directory

$ guix build -n openmpi
12.5 MB would be downloaded:
   /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2
   /gnu/store/7hizrpdsqf6q3pjgzmi51r5vbzlijkw0-python-minimal-wrapper-3.8.2
--8<---cut here---end--->8---


Cheers,
simon

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

>> does “guix build  -n” fit your use-case?
>
> Checking... yes! I hadn't even considered using "build" when building is
> exactly what I do not want to happen. But yes, it works just fine.

Not quite:

$ guix build -n zziplib
substitute: 
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
24,6 MB would be downloaded:
   /gnu/store/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i
   /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
   /gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
   /gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2
   /gnu/store/7hizrpdsqf6q3pjgzmi51r5vbzlijkw0-python-minimal-wrapper-3.8.2

My first impression is that "guix build -n" shows the store path of the
package only if all dependencies are in the store. Otherwise, it shows
the dependencies.

>> Well, ’package-output’ in (guix packages) is what you need, I guess.
>
> And that works just fine as well.

That one is worse because it even fetches/builds the dependencies:

$guix repl
GNU Guile 3.0.5
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> (use-modules (guix packages) (guix store) (gnu packages 
compression))
scheme@(guix-user)> (with-store store (package-output store zziplib))
substitute: 
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
fetching path `/gnu/store/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i'...
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Downloading 
https://ci.guix.gnu.org/nar/lzip/9h9153akbd6g2520mcl2d0vgbvyfi5qm-openssl-1.1.1i...
 openssl-1.1.1i  2.1MiB   3.2MiB/s 00:01 [##] 100.0%

fetching path 
`/gnu/store/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2'...
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
Downloading 
https://ci.guix.gnu.org/nar/lzip/fbn395nfpbp4d4fr6jsbmwcx6n10kg16-python-minimal-3.8.2...
 python-minimal-3.8.2  11.9MiB2.5MiB/s 00:05 [##] 100.0%
...


After that, I can do

$ guix build -n zziplib
/gnu/store/fx0cdzzppd8jc09sianbq6gl1h7mxx3x-zziplib-0.13.72


Cheers,
  Konrad.

Re: Finding the store path of a package

[Konrad Hinsen]

Hi Simon,

> does “guix build  -n” fit your use-case?

Checking... yes! I hadn't even considered using "build" when building is
exactly what I do not want to happen. But yes, it works just fine.

> Well, ’package-output’ in (guix packages) is what you need, I guess.

And that works just fine as well.

Thanks,
  Konrad.

Re: Finding the store path of a package

[zimoun]

Hi Konrad,

On Wed, 17 Mar 2021 at 18:55, Konrad Hinsen  wrote:

> I wonder if there is a straightforward way to find the store path
> corresponding to a package, assuming that the package actually is in the
> store. I don't care if it's done via the CLI or via Guile code.

does “guix build  -n” fit your use-case?


> but it also downloads/builds the package if it's not yet in the store,
> which is not what I want. In fact, I don't care what happens then the
> package is not in the store. Returning a non-existing path is fine,
> as is raising an error or returning #f.

Well, ’package-output’ in (guix packages) is what you need, I guess.

--8<---cut here---start->8---
$ guix gc -D $(guix build hello)
0.1 MB will be downloaded:
   /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
substituting /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10...
downloading from 
https://ci.guix.gnu.org/nar/lzip/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10 ...
 hello-2.10  51KiB  
454KiB/s 00:00 [##] 100.0%

finding garbage collector roots...
[0 MiB] deleting '/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10'
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 20885.94 MiB

$ guix repl
GNU Guile 3.0.5
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(gnu packages base) ; hello
scheme@(guix-user)> ,use(guix packages) ; package-output
scheme@(guix-user)> ,use(guix store) ; with-store
scheme@(guix-user)> (with-store store (package-output store hello))
$1 = "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
scheme@(guix-user)> ,q

$ ls /gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
ls: cannot access '/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10': No 
such file or directory
--8<---cut here---end--->8---



Hope that helps,
simon