Re: Reproducing a Python project environment (using guix inferiors)

[Brandon Ellington]

Thanks again for the reply simon!

zimoun  writes:

> Hi,
>
> On Mon, 30 Nov 2020 at 14:39, Brandon Ellington  wrote:
>
 | package of interest | guix commit  | status |
 |-+--+|
 | python-matplotlib   | "7e06086522" | bad|
 | python-pandas   | ce2cfcabfc   | bad|
 | python-networkx | 269f100330   | good   |
 | python-numpy| 4d6ed794dd   | bad|
 | python-scipy| 02ddafef55   | good   |
>
> Which version of  do you need?
You can kind of get it from [2] in my initial email, but here is a
cleaner table:

| package of interest | version (rec. minimum) |
|-+|
| python  |  3.6.5 | do not rebuild
|-+|
| python-matplotlib   |  2.0.2 | used to exist in guix
| python-pandas   | 0.22.0 |
| python-networkx |2.1 |
| python-numpy| 1.13.3 |
| python-scipy|  1.0.1 |
|-+|
| python-pygraphviz   |1.3 | bump from current guix
| python-nltk |  3.2.5 |
|-+|
| python-wikipedia|  1.4.0 | importing
| python-toposort |1.5 |
| python-community|   0.10 |
| python-louvain  |   0.14 |
|-+|
| (author's own)  |   (not applicable) |

> Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
> (Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
> «Inferior».  Therefore, they are not reachable by this mechanism.
> That’s one of the motivations for the channel “guix-past”. :-)
> 
> [...]
> 
> For commits before ~July 2018, the strategy is the one of ‘guix-past’
> channel; which I roughly described above.
>
> 1: 
> https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/

Ah, lesson learned! I'll direct my attention toward guix-past then :)

> For commits after the complete introduction of Inferiors mechanism
> (~July 2018, please read [1]), it is ok.  Be aware that each inferior
> has a cost and if your package foo depends on X packages coming from
> inferiors, then it means you are running under the hood X time: ‘guix
> pull’ then ’guix build ’ which could require to also build
> other dependencies, therefore installing foo could be long.

Indeed I noticed this. Building the two good channels was not too bad,
but I have not tried building networkx or scipy yet. As for your
hypothetical...

> Explicitly, imagine that python-foo depends on python, python-bar and
> python-baz.  Imagine that you use 2 inferiors, one for python-bar and
> one for python-baz.  Now imagine that python-bar and python-baz depends
> both on the package python.  Then, there is a high probability to build
> 3 times tiny variants of the package python.  And maybe the 3 packages
> python-foo, python-bar and python-baz perfectly works with the same
> variant of python.  And it is worse because that applies to all the
> implicit dependencies (compilers and so on).

...I can imagine it, and that would be terrible yes. I thought python
wouldn't be a dependency of python packages in part for that reason
though. Or maybe I read this for something else like emacs and
extrapolated to python. Maybe I'm dreaming :P. In any case, the overall
idea makes sense, especially the implicit dependencies thing. For me, if
I went for full reproducibility including the older version of python,
then all these packages would work on 3.6.5, and I'd still have a lot to
learn about the implicit dependencies.

> What I will do is: a) fix a Guix version b) create a channel containing
> all the necessary variants (backporting dependencies if required, i.e.,
> copy/pasting old package definition and fix them if they does not build
> anymore) to build my short list of .
>
> This is more or less the strategy used to feed the channel “guix-past“.
>
> And I will not use the inferior mechanism because it adds a lot of
> complexity and will not solve your problem since old packages are not
> reachable and/or your need to add old dependencies.

Got it. The last part of the last sentence is surprising (because the
mechanism must have been the right choice to someone/for something) but
everything else adds up.

> The time-machine is orthogonal with the way to distribute, IMHO.  It is
> simply a easy CLI to fix the Guix version.
>
> Usually, I do:
>
>   guix describe -f channels > channels.scm
>
>   edit my-manifest.scm # containing 
>   
>   [.. hack my custom variants ..]
>   [.. backport variant dependencies when I need for these variants ..]
>   guix time-machine -C channels.scm
>-- build -L /path/to/variants/ -m my-manifest.scm
>   [.. loop hack until it

Re: Avoiding PYTHONPATH - latest?

[Phil]

zimoun writes:

> You might be interested by the Efraim’s talk «Just build it with Guix»
> from the online Guix Day:

Yes this is extremely useful - thanks!

Re: How do you write a package's deps to a file?

[Bonface M. K.]

Hi all!

Was finally able to hack something that
works! See below:

zimoun  writes:
>
> (And I am still interested by the answer of «is it possible to get the
> metadata of the inputs in phases?» :-))
>

Yes it's possible. The trick is to use
"this-package" which can *only* be used inside a
package definition and no where else. Stumbled
across that when poring through some guix package
definitions, moreso skriblio.

Here's my working phase:
--8<---cut here---start->8---
(add-after 'install 'generate-dependency-file
 (lambda* (#:key inputs outputs #:allow-other-keys)
   (call-with-output-file
   (string-append
(assoc-ref outputs "out")
"/lib/python3.8/site-packages"
"/wqflask/DEPENDENCIES.md")
 (lambda (port)
   (format
port "
 System Inputs (generated from Guix)
|Name | Version | Home Page | Description |
|-|-|---|-|
|~a|
"
,(apply
  string-append 
  (map
   (lambda (input)
 (let* ((pkg (cadr input))
(name (package-name pkg))
(version (package-version pkg))
(home-page (package-home-page pkg))
(description (package-description pkg)))
   (string-append
"| **" name "** | "
version " | "
home-page " | "
description " | "
" |\n")))
  (package-propagated-inputs this-package)
 )))
--8<---cut here---end--->8---

-- 
Bonface M. K. 
Chief Emacs Bazu / Rieng ya software sare
Mchochezi of:  / Twitter: @BonfaceKilz
GPG Key: D4F09EB110177E03C28E2FE1F5BBAE1E0392253F


signature.asc
Description: PGP signature

Re: Workflow with mu4e + magit for sending patchsets to guix?

[zimoun]

Hi Pierre,

On Tue, 01 Dec 2020 at 10:10, Pierre Neidhardt  wrote:

> So here is my final, hopefully fully-functioning version then:

I am testing it.  Thanks!

Cheers,
simon

Re: Avoiding PYTHONPATH - latest?

[zimoun]

Hi,

On Tue, 01 Dec 2020 at 13:28, Phil  wrote:

> I could instead run my unit tests against my git clone, but then you're
> not testing what you will deliver, you're testing the source code that
> will be packaged, not the package itself.  The unit tests may fail if
> run on the package itself.
>
> The way around this in Guix, I imagine, is to rebuild a Guix package
> containing your own Python code each time you want to test it and then
> have the unit tests run as part of that package definition.

You might be interested by the Efraim’s talk «Just build it with Guix»
from the online Guix Day:

  

Roughly speaking, Guix allows an undocumented file usually named
“guix.scm” or “.guix.scm“ where the point is to easily do the “build and
test what is delivered” part.


Thanks,
simon

Re: Avoiding PYTHONPATH - latest?

[Phil]

Thanks again zimoun for your comments.

zimoun writes:


> I am not sure to understand what you mean.  Installing always means
> “fixed at package@version”.  I should miss something with your
> workflow.

So using pip in 'editable' mode installs your git clone via softlinks as a 
package.
It doesn't really have a version - it's the code you're currently
working on and will change every time you save a file without having to 
re-package.

There is no equivalent to this in Guix - by design, of course.

> This is a bad practise and should not be encouraged.  All in all, you
> end up with the big mess that Guix is trying to fix.

What I like about 'pip install -e' is that you are testing the actual
package that end-users are going to install - without the hassle of
explicitly installing it.

I could instead run my unit tests against my git clone, but then you're
not testing what you will deliver, you're testing the source code that
will be packaged, not the package itself.  The unit tests may fail if
run on the package itself.

The way around this in Guix, I imagine, is to rebuild a Guix package
containing your own Python code each time you want to test it and then
have the unit tests run as part of that package definition.

This is what I'm going to try to do, but it might end-up being a fairly
custom setup which may not integrate so well.


>
> For example, I am using Emacs and before I was using ’use-package’ which
> allows to locally tweak the packages that I depend on.  Then, I wanted
> to do the same with Guix.  At the end, it is wrong.  The right thing is
> inspect the package and if you need to fix, do “guix build -S ”,
> fix and create a variant using package transformation, for instance.
> This way, you have something reproducible, easy to share and/or deploy.
> IMHO.

Yes this makes total sense when I'm installing someone else package
rather than developing my own using pip.


> We discussed at the recent Guix Days that documentation “Getting
> starting with X” is lacking–where X is Python, R, C, Haskell, etc.

Yes this would be great - I'm happy to chip away at it, and even change
my setup to make it more Guix-sensible but as a launch-pad to get
people setup quickly with an explicit list of do this, avoid this -
would be great!

>
> Feel free to send a Cookbook recipe [1] about your Python setup. :-)
>
> 1: 

Will keep this in mind - once I have my setup more Guix-complaint.  I'd
be keen to get feedback on it too - as per my last e-mail the Python
community have a wide range of development setups, so it will probably
be community effort to write a cookbook that caters to, or at least
rationalises the various approaches that work in Guix.

Re: Avoiding PYTHONPATH - latest?

[zimoun]

Hi,
On Mon, 30 Nov 2020 at 15:04, Phil  wrote:

> I 100% agree it's a bad idea to mix packagers, but pip install's use-cases go
> beyond the need to do fixed installs of package@version, provided by
> Guix.

I am not sure to understand what you mean.  Installing always means
“fixed at package@version”.  I should miss something with your
workflow.


> Given that Guix profile's can't be edited from within a Guix profile
> (for obvious reasons), if I want to edit my code whilst it is installed,
> I see the only way to do this is by having a virtual env on top of a
> Guix profile that contains only my own module - with the Guix profile
> holding Python and all the imports I depend on.

This is a bad practise and should not be encouraged.  All in all, you
end up with the big mess that Guix is trying to fix.

For example, I am using Emacs and before I was using ’use-package’ which
allows to locally tweak the packages that I depend on.  Then, I wanted
to do the same with Guix.  At the end, it is wrong.  The right thing is
inspect the package and if you need to fix, do “guix build -S ”,
fix and create a variant using package transformation, for instance.
This way, you have something reproducible, easy to share and/or deploy.
IMHO.


> Perhaps the simple answer here is never mix pip and guix, and setup your
> development environment so there is never a requirement on pip.  But this is
> a limitation to people used to working this way.

My point of view is use pip or Guix, not both.  I was using Conda to
manage my Python stuff.  And same, I wanted to reproduce my Conda habits
with Guix.

Yeah, Guix changes the working habits, and as any changing habits, we
often do not like at first.  But, somehow it is only an habit.  And as
sugar in coffee, you need 3 weeks of complaints and then you will even
forget the taste of sugar. ;-)


> If nothing else, personally, it was a good off-the-deep-end way to get under 
> the
> covers of Guix's treatment of python and python packages :-)

We discussed at the recent Guix Days that documentation “Getting
starting with X” is lacking–where X is Python, R, C, Haskell, etc.

Feel free to send a Cookbook recipe [1] about your Python setup. :-)

1: 


All the best,
simon

Re: Reproducing a Python project environment (using guix inferiors)

[zimoun]

Hi,

On Mon, 30 Nov 2020 at 14:39, Brandon Ellington  wrote:

>>> | package of interest | guix commit  | status |
>>> |-+--+|
>>> | python-matplotlib   | "7e06086522" | bad|
>>> | python-pandas   | ce2cfcabfc   | bad|
>>> | python-networkx | 269f100330   | good   |
>>> | python-numpy| 4d6ed794dd   | bad|
>>> | python-scipy| 02ddafef55   | good   |

Which version of  do you need?


>> What do you mean by “bad”?
>
> For , guix has a definition for it at commit
> . When I declare an inferior channel at  and
> try to build that channel, I note in the  column whether it
> builds ("good") or I got the [3] message following a backtrace ("bad").
> If it were to build, then I would use (first (lookup-inferior-packages
> (<>-inferior))) in a manifest for the
> package itself.

Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
(Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
«Inferior».  Therefore, they are not reachable by this mechanism.
That’s one of the motivations for the channel “guix-past”. :-)

Basically, the Backtrace should say something as:

--8<---cut here---start->8---
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
error: %guix-register-program: unbound variable
--8<---cut here---end--->8---


>> What did you run?  The command and the file(s)?
> I attached the file I used to define the channels and inferiors. As for
> the command, I just load these definitions piecemeal into the repl.

I see.

>>> 619:8  4 (_ #(#(#(#) "/gnu…" …) …))
>>>626:19  3 (_ #(#(#(#) "/gnu…" …) …))
>>> 155:9  2 (_ #(#(#(#) "/gnu…" …) …))
>>>223:20  1 (proc #(#(#(#) "/…" …) …))
>>> In unknown file:
>>>0 (%resolve-variable (7 . %guix-register-program) #)
>>> #+end_example

Ah, that’s the same backtrace message showed above. :-)


> I do not think so. I have three different scenarios I'm working with:
> 1. python-foo does not exist in guix :: So I use guix import to give me
> a package definition and work from there (so far this is okay, but if it
> weren't because of its dependencies I see where we might need to follow
> your hypothetical).
> 2. python-foo does exist in guix, but the desired version is in the past
> :: So I (attempt to) use a guix inferior from a channel where it was
> defined (this is where I have issues, but I assume that if guix had a
> definition for it, then python-bar and python-baz would both be
> available in the entire commit range that it defines python-foo).
> 3.
> python-foo does exist in guix, but I need a newer version :: I inherit
> from the old version, changing the necessary fields (like 1. I see where
> we would need to follow the hypothetical, but I haven't seen any missing
> dependencies errors).
>
> I successfully made the channel https://github.com/branjam4/cna-python
> (with packages from 3. and 1.) when I last worked on this. Hopefully
> having a successful channel build isn't deceiving me, but I thought
> given the successful build that I got lucky with needing to figure out
> dependencies. My primary block is this "unregisted program" error I get
> when using guix inferiors that I'd like to add to my manifest.

What I will do is: a) fix a Guix version b) create a channel containing
all the necessary variants (backporting dependencies if required, i.e.,
copy/pasting old package definition and fix them if they does not build
anymore) to build my short list of .

This is more or less the strategy used to feed the channel “guix-past“.

And I will not use the inferior mechanism because it adds a lot of
complexity and will not solve your problem since old packages are not
reachable and/or your need to add old dependencies.


>> From my experience, one simple way to start is:
>>
>>   guix time-machine --commit= \
>>-- build -L  
>>
>> where  is an old Guix commit providing bunch of dependencies used
>> by  of interest defined in the folder
>> .
>> 
>> A channels.scm file with the option ’--channels/-C’ seems even better.
>
> Hmm, my end goal is to make /something/ that will be simple for
> people with minimal assumed experience in guix (my professor, for
> example) to run in a virtual machine with guix and direnv available, so
> they can pull a git repository, then follow code examples from the book
> without focusing on dependency building (there would be the initial cost
> of waiting for things to build the first time, though). Assuming
> time-machine works, is it straightforward to make a profile from it (or
> something else enabling direnv to understand what I want when I go into
> the cna-python directory)? That was something I missed while reading the
> time-machine part of the manual.

The time-machine is orthogonal with the way to distribute, IMHO.  It is
simply a easy CLI to fix the Guix version.

Usually, I do:

  guix describe -f channels > channels.scm

Re: How do you write a package's deps to a file?

[zimoun]

Hi,

On Tue, 01 Dec 2020 at 12:13, "Bonface M. K."  wrote:

>> What do you want?  Get the list of inputs and their metadata when typing
>> “guix install foo”?  And this list becoming part of the outputs?  Or
>> simply get somehow the list of inputs and their metadata?  Which is
>> straightforward.
>>
>
> Yes! Ideally I want to get details of the package
> inputs, so that I could use that info to write to
> a file somewhere in the outputs dir. The end goal
> is to use that file to display (on some website)
> the dependencies the project has. So when you do,
> say, `guix install `, as part
> of your outputs, you also generate a file that
> explicitly list the deps as part of your output ;)
>
>>
>> Well, I am interested by the answer of: is it possible to get the
>>
>
> I think Efraim's suggestion in this thread is more
> pragmatic ;)

The Efraim’s suggestion–also the Julien’s suggestion about use the
package data directly– is not what you are describing above.  It is
“simply get somehow the list of inputs and their metadata, which is
straightforward.” :-)

Basically, first step you run “./pre-inst-env guix repl -- script.scm”
which contains ’package-inputs’ (or any other ’package-direct-inputs’,
’package-propagated-inputs’, etc.) with formatting and second step you
manually add to the package this generated file in one phase to get it
in the output.

The file ’script.scm’ is more or less one illustration in this blog
post:




Now, is it possible to embed this first step in the package definition
as an extra phase and run it via ’invoke’ or similar?


(And I am still interested by the answer of «is it possible to get the
metadata of the inputs in phases?» :-))

All the best,
simon

Re: How do you write a package's deps to a file?

[Bonface M. K.]

Hi Efraim!

Efraim Flashner  writes:

[...]
>> Notice that the version, home page and description
>> are repeated across rows. Herein lies my question:
>> How would you fetch the details of the inputs(like
>> version, home-page, description) etc etc from a
>> modifed phase. Do I need to create a closure from
>> which I can import the modules like say _(use (gnu
>> packages))_. How do I do that if that's the case?
>> 
>> Is it also possible to get the same information in
>> a fine grained manner, like say get the
>> propagated-inputs instead of getting /all/ inputs?
>> 
>
> I think it would be easiest to compute this outside the package
> definition and then add it as an input and just plop the precomputed
> file into place in a custom phase.

Yes! I think this will do it for this case. Would
never have thought of that! Thanks!

-- 
Bonface M. K. 
Chief Emacs Bazu / Rieng ya software sare
Mchochezi of:  / Twitter: @BonfaceKilz
GPG Key: D4F09EB110177E03C28E2FE1F5BBAE1E0392253F


signature.asc
Description: PGP signature

Re: How do you write a package's deps to a file?

[Bonface M. K.]

Hi!

zimoun  writes:

[...]
> …I am not convinced it is the right level to extract the
> information you need since ’input’ is, e.g.,
>
>   ;;; (("source" . 
> "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz"))
>   ;;; (("grep" . "/gnu/store/yhqc5imq7sn2sf9f48dj4gk12y4kw4s4-grep-3.4"))
>   ;;; (("libc:static" . 
> "/gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static"))
>   
> therefore you need ’specification->package’ to find back the package
> mapping to the string in order to access the metadata (version,
> home-page, etc.).  Aside the issue that some do not have corresponding
> packages (source, libc:static for instance).  Aside the issue of
> multiple versions, for instance compare the version of the ’gcc’ input
> of ’hello’ returning the string “gcc” with:
>
> scheme@(guix-user)> (specification->package "gcc")
> guix repl: package 'gcc' has been superseded by 'gcc-toolchain'
> $1 = # 7f3b72596640>
>

I sse that now. Thanks for pointing that out.

>
> What do you want?  Get the list of inputs and their metadata when typing
> “guix install foo”?  And this list becoming part of the outputs?  Or
> simply get somehow the list of inputs and their metadata?  Which is
> straightforward.
>

Yes! Ideally I want to get details of the package
inputs, so that I could use that info to write to
a file somewhere in the outputs dir. The end goal
is to use that file to display (on some website)
the dependencies the project has. So when you do,
say, `guix install `, as part
of your outputs, you also generate a file that
explicitly list the deps as part of your output ;)

>
> Well, I am interested by the answer of: is it possible to get the
>

I think Efraim's suggestion in this thread is more
pragmatic ;)

> All the best,
> simon
>

-- 
Bonface M. K. 
Chief Emacs Bazu / Rieng ya software sare
Mchochezi of:  / Twitter: @BonfaceKilz
GPG Key: D4F09EB110177E03C28E2FE1F5BBAE1E0392253F


signature.asc
Description: PGP signature

Re: Workflow with mu4e + magit for sending patchsets to guix?

[Pierre Neidhardt]

Ah!  That's good to know!

So here is my final, hopefully fully-functioning version then:

--8<---cut here---start->8---
(defvar ambrevar/known-mailing-list-archives
  '(("help-guix@gnu.org" . "guix")
("guix-de...@gnu.org" . "guix")
("debbugs.gnu.org" . "guix"))
  "Alist of mail adresses and their Yhetil name.
Alternatively the key may just be a host name against which a
recipient will be matched.")

(defun ambrevar/guess-yhetil-link (message-id)
  (let* ((all-addresses (mapcar #'second
(mail-extract-address-components
 (mapconcat #'identity
(list
 (notmuch-show-get-header :To)
 (notmuch-show-get-header :Cc))
", ")
 'all)))
 (match-address (lambda (address-or-host)
  (if (string-match "@" address-or-host)
  (member address-or-host all-addresses)
(seq-find (lambda (address)
(string-match address-or-host address))
  all-addresses
 (mailing-list (alist-get
(seq-find match-address
  (mapcar #'car 
ambrevar/known-mailing-list-archives))
ambrevar/known-mailing-list-archives
nil nil #'string=)))
(when mailing-list
  (concat "https://yhetil.org/";
  mailing-list "/" message-id

(add-to-list 'notmuch-show-stash-mlarchive-link-alist
 (cons "Yhetil" #'ambrevar/guess-yhetil-link))

(setq notmuch-show-stash-mlarchive-link-default "Yhetil")
--8<---cut here---end--->8---

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: How do you write a package's deps to a file?

[Bonface M. K.]

Hi!

Julien Lepiller  writes:

> The reason you always get the same data is because
> you're assigning "name" inside the sexp
> (build-side), but you're using it inside tge
> unquote (host-side), so it refers to the name of
> the package you're building.
>

Ah yes! I see that now. Silly me ~.~

> I think you're doing it wrong, because some input
> names have no corresponding package (see source
> for instance). Why not use the package data
> directly, like guix graph does for instance?
>

I don't know how to do this. I've had a look at
guix/graph.scm and scripts/graph.scm... And I got
lost--- I'm not /yet/ a scheme guru... I reckon
I'll go with Efraim's suggestion which seems
easier to do :)

Thanks for the pointers!

[...]
-- 
Bonface M. K. 
Chief Emacs Bazu / Rieng ya software sare
Mchochezi of:  / Twitter: @BonfaceKilz
GPG Key: D4F09EB110177E03C28E2FE1F5BBAE1E0392253F


signature.asc
Description: PGP signature