Re: Guile 3 migration plan
Hello Guix!
Ludovic Courtès skribis:
> 2. We must be able to run Guix itself on Guile 3. This is already
> possible if you add “3.0” to the ‘GUILE_PKG’ invocation in
> ‘configure.ac’, but there are still a few test failures that I’ve
> been working on. Almost all of them relate to the use of ‘@@’ and
> ‘mock’, which do not work with Guile 3 “declarative modules”.
>
> In many cases, it’s an opportunity to clean up our tests; see for
> example 4aea90b1876179aab8d603a42533a6bdf97ccd3c and the preceding
> commit. It should be possible to be done real soon!
Good news everyone! It builds and all the tests pass since
7b2a47a702b7393cd968640079f8703c932d1405, and commit
da7651806102d637253cb9f5677b96d6a178fc05 adds a “guile3.0-guix” package!
> Once we’re done, we can change (guix self) so that ‘guix pull’
> pulls Guix on Guile 3.
The attached patch does that, I tested it with “make as-derivation”.
I’ll do some more testing but unless I’m overlooking something, we can
apply it anytime now.
Ludo’.
diff --git a/guix/self.scm b/guix/self.scm
index 207e80d842..f883f14a2c 100644
--- a/guix/self.scm
+++ b/guix/self.scm
@@ -49,13 +49,13 @@
(let ((ref (lambda (module variable)
(module-ref (resolve-interface module) variable
(match-lambda
- ("guile" (ref '(gnu packages commencement) 'guile-final))
- ("guile-json" (ref '(gnu packages guile) 'guile-json-3))
- ("guile-ssh" (ref '(gnu packages ssh) 'guile-ssh))
- ("guile-git" (ref '(gnu packages guile) 'guile-git))
- ("guile-sqlite3" (ref '(gnu packages guile) 'guile-sqlite3))
- ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile-gcrypt))
- ("gnutls" (ref '(gnu packages tls) 'gnutls))
+ ("guile" (ref '(gnu packages guile) 'guile-3.0))
+ ("guile-json" (ref '(gnu packages guile) 'guile3.0-json))
+ ("guile-ssh" (ref '(gnu packages ssh) 'guile3.0-ssh))
+ ("guile-git" (ref '(gnu packages guile) 'guile3.0-git))
+ ("guile-sqlite3" (ref '(gnu packages guile) 'guile3.0-sqlite3))
+ ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile3.0-gcrypt))
+ ("gnutls" (ref '(gnu packages tls) 'guile3.0-gnutls))
("zlib" (ref '(gnu packages compression) 'zlib))
("lzlib" (ref '(gnu packages compression) 'lzlib))
("gzip" (ref '(gnu packages compression) 'gzip))
@@ -1121,9 +1121,9 @@ is not supported."
version))
(define guile
-;; When PULL-VERSION >= 1, produce a self-contained Guix and use Guile 2.2
-;; unconditionally.
-(default-guile))
+;; When PULL-VERSION >= 1, produce a self-contained Guix and use the
+;; current Guile unconditionally.
+(specification->package "guile"))
(when (and (< pull-version 1)
(not (string=? (package-version guile) guile-version)))
@@ -1142,7 +1142,7 @@ is not supported."
(shorten version))
#:pull-version pull-version
#:guile-version (if (>= pull-version 1)
- "2.2" guile-version)
+ "3.0" guile-version)
#:guile-for-build guile)))
(if guix
(lower-object guix)
Re: Guile 3 migration plan
Hi! Vagrant Cascadian skribis: > On 2020-01-16, Ludovic Courtès wrote: >> Today is Guile 3 day, a day to celebrate, woohoo! \o/ > > Yay! > >> How will Guix migrate to Guile 3? That’s a threefold process: > ... >> 2. We must be able to run Guix itself on Guile 3. This is already >> possible if you add “3.0” to the ‘GUILE_PKG’ invocation in >> ‘configure.ac’, but there are still a few test failures that I’ve >> been working on. > ... >> Once we’re done, we can change (guix self) so that ‘guix pull’ >> pulls Guix on Guile 3. > > Will it still be buildable from source with only guile 2.2? I have no > idea when Debian will have guile 3 available, for example, and was > hoping to be able to start the process of uploading ... "soon". For some time (a few months at least?) we’ll make sure Guix can be built from source with both 2.2 and 3.0. Ludo’.
Re: Inverted index to accelerate guix package search
> What is not clear to me right now in both implementations are. > > 1. > How to update the index. > Give a look at the "pull" code and the ~/.cache/guix folder. We don't "update" the index. At every guix pull we create it anew. Currently, generate-package-cache in gnu/packages.scm does this. generate-package-cache is called by package-cache-file in guix/channels.scm. package-cache-file is a channel profile hook listed under %channel-profile-hooks. Now, what I am unclear about is how to test my sqlite index building code without actually pushing to master and running a guix pull. I will go through the various tests in Guix and see if I can figure something out, but any pointers would be much appreciated. > 2. > How to deal with regexp. > It is more or less clear to me how to deal with using the trigram keys > but I do not know with SQLite; I have not thought about yet. I think it is not possible to search using regular expressions in sqlite unless some external module is loaded. See https://stackoverflow.com/questions/5071601/how-do-i-use-regex-in-a-sqlite-query/8338515#8338515 I think we should remove regex support altogether. I don't think a good search interface should expect the user to provide regexes for search. Certainly, it will be a lot less useful if and when we have xapian. However, just to keep backward compatibility, we can fall back to brute force fold-packages search for regexes. As Ludo pointed out, we can't remove the brute force code since we need to support cases when the cache is not authoritative. > If you want to implement it, go ahead. :-) Yes, I'll give it a shot. :-) I have some other commitments over the weekend, but hopefully I'll have something by Monday night. > Otherwise, I will try to finish next week what I started yesterday > evening using VHash. :-) About sqlite versus an inverted index using vhashes, I don't know if it is possible to serialize a vhash onto disk. Even if that were possible, we'll have to load the entire vhash based inverted index into memory for every invocation of guix search, and that could hit performance. Something like guile-gdbm could have helped, but that's another story. Also, I now agree with your earlier assessment that we should delegate all this to sqlite. :-) That guix already uses sqlite for other things is all the more reason. > (note that to avoid duplicate , the file sets.scm can be relevant) I didn't know about sets.scm when I wrote my first proof of concept inverted index script. That is why I reinvented the set using hash tables. I don't know how hash tables are different from VHashes or which is better. Cheers! :-) Arun. signature.asc Description: PGP signature
Re: Inverted index to accelerate guix package search
Oh, and is there a Guile library to write SQL statements in a more lispy way? E.g. like https://github.com/fukamachi/sxql. -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Inverted index to accelerate guix package search
Arun Isaac writes: > For the time being, since we don't have xapian bindings, I think we > should settle for sqlite's full text search capabilities. > > https://www.sqlite.org/fts5.html > > I have attached a short proof of concept script for an sqlite based > search. Speedup is around 200x, and populating the database only takes > around 2.5 seconds. Here is a sample run. > > Sqlite database populated in 2.5516340732574463 seconds > Brute force search took 0.11850595474243164 seconds > Sqlite search took 5.459785461425781e-4 seconds This is really cool! And quite simple too! So now I suppose the test would be to try with some real world examples :) I don't know the kind of tests we can write for this though. If the results are convincing enough, I'd agree with you: we can first settle for SQLite before moving to something more sophisticated like Xapian. Cheers! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Parameterized packages
Hi Zimoun! zimoun writes: > Well, it would ease comparison in the HPC world. :-) > > > It is not related to the "parametrized package" in the sense of flag > options. :-) > > And I do not know if it make sense. What do you think? I think you are making a lot of sense and yes, parameters should probably be able to customize the builder as well. But as I understand it, all the "configurable part" of the builder is expose through the #:arguments field. A simple example of this is python: the python-build-system has an argument which allows the package to specify whether we use python-2 or python-3. In this case, it's trivial to use parameters to influence which compiler the build system will use. For gnu-build-system (with gcc, clang, etc.) we can probably do similar things already by setting CC. The solution to your problem in my opinion is simply to expose just the right amount of options through #:arguments for all build systems. Would that be satisfactory to you? -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Parameterized packages
Hi! L p R n d n writes: > (define-modificator headless > `((xorg (modify-package xorg > ...)) Thanks for sharing, this is an interesting approach! But it may raise a major challenge. The point of parameters is that they must compose (e.g. headless + static-lib), so can we do that with the "modificator" approach? Say for instance that 2 parameters in particular are not compatible for a given package, how to we handle that? This probably needs more thoughts! :) -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Parameterized packages
Hi Ricardo, Ricardo Wurmus writes: > I think this is a sensible approach, though it would require agreement > or at least coordination among package contributors about what > parameters to use. For example, one such parameter could be > “#:headless” to disable any graphical user interface or desktop feature, > but a case could just as well be made for “#:gui” or “#:graphical”… > > Coordination hasn’t been a problem so far because we managed to agree on > build systems and utilities, but it seems to me that parameters to match > on span a much larger range — I feel reminded of the problem of package > tags or categories, which are a similarly large field that defies > agreement. I think Tobias suggestion is a good solution to this: make parameters first class, global objects. A package can only use globally declared parameters. This way we have a limited set of well-defined parameters under control (understand: no duplicates, consistent naming). As Tobias pointed out, global declaration of parameters allows us to centralize their documentation and types. -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Parameterized packages
Hi ison!
ison writes:
> Maybe the current discussion is trying too hard to emulate Gentoo's USE
> flags and dependency graph concept (perhaps its my fault for bringing up
> global flags). But that feels like introducing "side effects", and maybe the
> whole idea should be treated more "functionally" in Guix.
>
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.
Yes, I am starting to come to a similar conclusion.
I think this makes more sense in terms of maintainability.
What that in mind, maintenance would still get more complex than it is
today: a package with N parameters would have 2^N possible outputs.
It quickly gets hard to test them all. That said, we can probably assume
that in most cases parameters won't conflict with each other.
--
Pierre Neidhardt
https://ambrevar.xyz/
signature.asc
Description: PGP signature
Re: Command line option in addition to GUIX_PACKAGE_PATH - Reason?
I have already merged, can you rebase your changes against master? :) -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Parameterized packages
On Thu, 16 Jan 2020 at 20:06, ison wrote:
> On Wed, Jan 15, 2020 at 02:54:25PM +0100, zimoun wrote:
> > And what I was thinking is a mechanism to easily set some arguments to
> > the build-system; for example changing the compiler toolchain (say
> > replacing GCC by Clang/LLVM).
> >
> > Well, as I said, I do not know if it is related to "parametrized
> > packages" because I am not sure to understand the final aim for these
> > "parametrized packages". :-)
[...]
> That is, simplify the problem to the mere concept of passing arguments to
> functions and nothing more. Take the headless server example: some parameter
> is passed to a package such as
> '("-X")
> That package would then be entirely responsible for what to do with them. If
> the package decides not to pass the same parameters to its inputs then the
> inputs are simply built without any parameters.
I agree.
What I was trying to suggest and/or discuss is to see the build-system
as a function where the compiler toolset is one argument.
Let consider these "compilers" (associated with complete toolset for
"compiling"): OCaml@4.01, OCaml@4.07, OCaml@4.09, CPython@2.7,
CPython@3.5, CPython@3.6, PyPy (not yet in Guix), GCC@8, GCC@9,
Clang@8, Clang@9, etc.
Today, for one particular build system, the compiler is fixed. To use
another compiler than the default one, the trick is to have
'package-with-explicit' and each build-system has to provide
'package-with-' where is hard coded.
To be clear, the seed (bootstrap) fixes how the compilers OCaml@4.01,
OCaml@4.07, OCaml@4.09, CPython@2.7, CPython@3.5, CPython@3.6, PyPy
(not yet in Guix), GCC@8, GCC@9, Clang@8, Clang@9, etc are compiled.
Then, it is difficult to use them to compile a package with one of
them.
Do we end with 'package-with-ocaml4.01', 'package-with-ocaml4.07',
'package-with-ocaml4.09', 'package-with-python2',
'package-with-python3.5', 'package-with-python3.6',
'package-with-pypy', 'package-with-gcc8', 'package-with-gcc9', etc.?
Do the users have to write all these 'package-with-'?
And if one wants to use CPython@3.5 and GCC@8, does they need to
define the package using:
(define-public my-foo
(package-with-python3.5
(package-with-gcc8 foo)))
?
I feel something is lacking.
And do not take me wrong, I am not suggesting that Guix should
maintain and ensure it works for all the combinations. The default is
already enough! :-)
And yes, it means potentially recompiling the world. Try to plot "guix
graph -t bag-emerged foo". ;-)
Why do I want that?
For example, a couple of software have been used to simulate complex
physical phenomena. Using the commit A -- which provides the seed A
and all compiling tools in the version X -- I get some results. Then
using the commit B -- which provided the seed B and all the compiling
tools in the version Y -- I get different results. Where does come
from the difference? From my couple of software which are not
reproducible? From round-off errors and floating point arithmetic
mess? Well, the most probable... but can I reduce the search space of
the difference? And I would like to be able to fix the seed to A, then
used the compiling tools in the version X and compare using this very
same seed A with the compiling tools in the version Y. Do the same
with seed B.
Well, it would ease comparison in the HPC world. :-)
It is not related to the "parametrized package" in the sense of flag
options. :-)
And I do not know if it make sense. What do you think?
All the best,
simon
Re: TeX Live 2019 wanted
Hey Marius, > Marius Bakke writes: > >> I'll look into updating the remaining texlive packages shortly. > > Now this was some rocket science... > > The attached 9 patches updates everything (I think) to 2019.3. This looks great! It’s odd that they removed the few remaining scripts for generating hyphenation patterns and replaced them with the output files, but it sure simplifies some of these package definitions. > The only regression I found from 'master' was that "discrover" fails to > find math fonts during the build process. Hmm, that’s a little worrying. I spent much more time than I think is reasonable to work on The Font Problem in LaTeX. This really shouldn’t break. I’m willing to investigate this, but it would be easier if we had a branch for this upgrade and had ci.guix.gnu.org build it all. I don’t want to build all these things on my laptop again. -- Ricardo
Re: Presentation BlueHats (french workshop)
Hi Ludo, Thank you for reviewing and pushing. If you have a tight slot of time, you could add the 2 missing files: talks/fosdem-2017/hpc/images/shrink-wrap.jpg talks/fosdem-2017/hpc/images/shrink-wrap2.png I think they are on your machine 'ribbon'. ;-) On Wed, 15 Jan 2020 at 23:07, Ludovic Courtès wrote: > It’s nice you were able to talk at POSS. I suppose the audience was not > necessarily familiar with reproducible science, right? To be exact, a subset of POSS. :-) Yes, they were not, at all! Interesting "exercise". :-) > > To guarantee Reproducible Science in the modern age of data, we need to > > guarantee several items, especially: > > 1. Open Articles > > 2. Open Data > > 3. Open Source > > 4. Controlled computing environment (open, too) > > Today, initiatives have been starting, to name some, about 1. > > [[http://rescience.github.io/][ReScience journal]] > > or french specific [[https://hal.archives-ouvertes.fr/][HAL]], 2. > > [[https://zenodo.org/][Zenodo]] and 3. > > [[https://www.softwareheritage.org/][Software Heritage]]. > > Yup! Not a fan of “open” which I find confusing here, but definitely a > fan of putting all this in perspective! Héhé! Currently, Open sounds to people; they have a vague concept in mind because all the marketing around. For example, the French archive says: "The open archive HAL". :-) Another example, ReScience uses "open" [1] and speaks about "open source community" [2]. [1] http://rescience.github.io/ [2] http://rescience.github.io/board/ Yet another example from Software Heritage [3]: Open Access repo, Open Data Sets repo, Open Source repo. [3] https://www.softwareheritage.org/mission/science/ Well, the message is: everything needs to be transparent; and Open is the vector to hook the message. I am not sure that enter in the technical debate about free / libre vs open is relevant when speaking about Reproducible Science. Philosophically speaking, the code used in Science needs only freedom 0 and freedom 1 to then claim be Reproducible Science, IMHO. And it is what I understand from the paragraph 'Proprietary Software' of your article [3]. ;-) [3] https://hal.inria.fr/hal-01161771v2/document The conclusion is: naming is hard! ;-) Hope to be able to discuss more on these topics here or there because they really matter. Thank you again. Cheers, simon
Re: Parameterized packages
Hello! Also followed the thread from a little far away. It's probably an awfull idea (I don't even know if it's possible) but throwing in different perspetives migh give birth to new ones so here I go. Instead of thinking in terms of use flags or parameterized packages maybe using something that I would call "modificators" might be an idea. I see it as some kind of list of funtions modifying packages (If it's clear?). The main differences would be that the package definitions would be left untouched and that modificators' definitions would be centralized keeping the complexity and the maintaining burden away. Something like modificators maintainer might emerge then? That could look like: (define-modificator headless `((xorg (modify-package xorg ...)) or something replacing a package directly without introducing new syntax: (xorg (package (inherit xorg) ... Then givin a list of packages, apply a modificator to it and get a modified list of packages? Hope it helps, Have a nice day, L p R n d n
Re: TeX Live 2019 wanted
Mikhail Kryshen writes: > - font files are missing from texlive-latex-lh (shouldn't it be called > texlive-lh?). Thank you, that’s a good one. Generally, the texlive-latex-* packages are of the old type that may very well be incomplete. I’ll replace this one with texlive-lh which will include all the source files as it should. Thanks for the report! -- Ricardo
