Re: A plan for parameterized packages

[Christopher Baines]

Ludovic Courtès  writes:

> Hi,
>
> Denis 'GNUtoo' Carikli  skribis:
>
>> The issue is that despite all that, the size of the images tend to
>> increase too rapidly over time[1].
>
> Yeah, that’s also the problem here: we have ‘guix size’ to profile a
> package at one point in time, but it’s easy to unwillingly increase its
> closure size the next day without noticing.
>
> Chris: does the Data Service track store item sizes (and more generally
> everything ‘query-path-info’ returns)?  It’d be great to be able to
> visualize size plots over time!

The Guix Data Service can store narinfos, which I believe contain all
the information you need to do something similar to what `guix size`
does. I believe you'd be able to track the size of individual items, as
well as closure size through the reference information.

So, yeah, I think it's definately possible to do what you describe.

Chris


signature.asc
Description: PGP signature

Re: A plan for parameterized packages

[zimoun]

Hi,

On Fri, 20 Nov 2020 at 12:40, Ludovic Courtès  wrote:

> > The issue is that despite all that, the size of the images tend to
> > increase too rapidly over time[1].
>
> Yeah, that’s also the problem here: we have ‘guix size’ to profile a
> package at one point in time, but it’s easy to unwillingly increase its
> closure size the next day without noticing.

I think that should be part of the tooling we need to ease the
"release process".  I mean, that's what I have tried to describe as
one starting point at the BoF discussion: tools that help to know how
is the shape of Guix at one point in time (via scripts and repl or via
commands and options) and then time-machine does the rest.

What is currently missing (from my little experience):

  - check reproducibility
  - check size
  - check substitutes per build system, list the "essentials" (the
ones we *absolutly* want to be substituable), maybe per groupes by
topics


> Chris: does the Data Service track store item sizes (and more generally
> everything ‘query-path-info’ returns)?  It’d be great to be able to
> visualize size plots over time!

Yeah, plot could be really useful.


All the best,
simon

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

Denis 'GNUtoo' Carikli  skribis:

> Last time I looked into how LibreCMC/OpenWRT did that, they had much
> more optimization than that. If I recall well, they use at least:
> - sstrip to strip binaries as much as they could. sstrip produces
>   smaller binaries than with strip.
> - compilation flags like -Os
> - a read-only compressed filesystem with an overlay to store the
>   changes

To me this looks like the ultimate size optimization level.  Before we
get there, we should first see how to get closer to package sizes
typically found on Debian and that alone is a real challenge.

> The issue is that despite all that, the size of the images tend to
> increase too rapidly over time[1].

Yeah, that’s also the problem here: we have ‘guix size’ to profile a
package at one point in time, but it’s easy to unwillingly increase its
closure size the next day without noticing.

Chris: does the Data Service track store item sizes (and more generally
everything ‘query-path-info’ returns)?  It’d be great to be able to
visualize size plots over time!

Ludo’.

Re: A plan for parameterized packages

[Denis 'GNUtoo' Carikli]

On Sun, 15 Nov 2020 22:24:29 +0100
raingloom  wrote:
> Alpine already achieves an incredibly tiny install size by splitting
> packages into many outputs. We could and should do the same.
> As far as I know, they do not have parameterized packages.
That also depends on how far you want to go.

Last time I looked into how LibreCMC/OpenWRT did that, they had much
more optimization than that. If I recall well, they use at least:
- sstrip to strip binaries as much as they could. sstrip produces
  smaller binaries than with strip.
- compilation flags like -Os
- a read-only compressed filesystem with an overlay to store the
  changes

The issue is that despite all that, the size of the images tend to
increase too rapidly over time[1].

If we manage to shrink Guix enough, it might be possible to use it on
way more devices, including RYF compliant devices or potentially
certifiable devices:
- The Talos II BMC has 32M according to both the wiki[2] and the image
  sizes[4]. Its architecture is ARM. So once we have the PPC64
  architecture working, it would be great to be able to run Guix
  both in the BMC and on the PowerPC CPU.

  That BMC is also available on other mainboards like the D16 which is
  supported by Libreboot, but the flash size is probably even smaller
  there.
- Many WiFi access point have very few flash space. It can boils down
  to as low as 16M for LibreCMC/OpenWRT compatible devices, or even 8M
  for older devices. However they typically use the MIPS architecture
  which isn't supported yet in Guix.
- There is a GNU/Linux distribution[6] that runs inside the flash chip
  where Libreboot or Coreboot typically runs. The goal is to enable
  more flexible and/or secure booting by using GNU/Linux to boot
  GNU/Linux. Here too the flash chip of computers supported by
  Libreboot can be quite small, like 8M for Thinkpads with GM45
  chipsets.

In some case it might be possible to increase the flash chip size
(sometimes you don't need soldering for that), but at least with x86
mainboards, the chipset has limits on the size of the flash chip that it
can see. And the size cannot be increased that much: The biggest flash
chip that flashrom supports is 256M.

References:
---
[1]https://openwrt.org/supported_devices/864_warning
[2]The wiki[3] mention a MX25L25635F/MX25L25645E/MX25L25665E flash chip
   which is 32M according to flashrom -L
[3]https://wiki.raptorcs.com/wiki/Debricking_the_BMC#Flash_new_BMC_firmware_via_serial_port_.28Open_Source_Method.29
[4]Once uncompressed the image[5] size (for installation through the
   shell) is 32M.
[5]https://wiki.raptorcs.com/wiki/File:Talos-ii-openbmc-v2.00-bundle.tar
[6]https://github.com/osresearch/heads/
[7]https://github.com/osresearch/heads/tree/master/config

Denis.


pgpoPCdagb2In.pgp
Description: OpenPGP digital signature

Re: A plan for parameterized packages

[Stephen Christie]

> Hi Pierre,

> Pierre Neidhardt  skribis:

> > One of the biggest struggle we had when discussing it was figuring out
> > what to do about parameter propagation across dependencies.

> > For instance, what if we want to build "all packages without X support"?
> > This means that the parameter must traverse all inputs recursively if we
> > don't want to drag X indirectly.

> > If I understand your change correctly, the patch is only applying
> > parameters to the given package and it's not propagated to the inputs,
> > is that correct?

> That’s correct: in this patch set parameters are per-package, and I
think it’s easier to start simple, but we could implement what you
describe without too much hassle I think.

> For example, you’d type ‘--with-parameters=x11=false’, and that’d be
applied to all the packages that have an ‘x11’ parameter.

> Ludo’.

I have done a lot of work with the Conan package manager, a c++ language 
package manager, that has grown in capability. It is not fully functional, but 
works on the hash of the key parameters of the package (name, version, etc.) to 
find the right reproducible binary. Two important parameters are "options" and 
"settings".

Options are per-package, and generally affect none below it. You can specify 
defaults for the options in the package, and also call for specific options on 
dependencies in package "recipes". There are also ways to define 
incompatibilities and substitutes. On the command line, you can specify options 
with -o,--options, with no namespace needed for the package you are installing, 
and package:option to specify for other packages pulled in. I prefer this 
syntax to all the equal signs you proposed (though I defer if this is standard 
throughout Guix/Guile)

Settings are more "system-wide", though being a language package manager, it 
does not have a "system". The same settings are applied to the whole tree 
during an install, and are usually things like compiler, architecture, and 
build type. These settings are chosen through a profile file, of which there is 
a default generated for a given computer. Settings can also be set at the 
command line during install with -s,--settings, but of course there is no 
namespacing.

https://docs.conan.io/en/latest/mastering/conditional.html

I think there is a lot of good stuff in Conan that Guix could learn from. It's 
a lot closer in architecture than any of the traditional system package 
managers.

If you would, please forward this to the mailing list that this message was 
from, as I am not currently on that list.

Sincerely,
Stephen Christie

Re: A plan for parameterized packages

[Stephen Christie]

Ludo,

The self.options["openssl"].shared does set the option for openssl, but this is 
not different than providing `-o openssl:shared` on the command line; it uses 
this to determine which package was actually requested. Conan packages are also 
non-configurable in the end, stored as the hash of it's name and parameters. 
This is probably done while calculating the tree before installing/building 
everything, but once that tree is calculated, it is fixed.

Besides parameters determining the hash of the package, 
OpenSSL/2.1@memsharded/testing and openssl/1.0.2u are most certainly separate 
packages. Conan has this name/version@username/channel structure to help find 
alternatives (semver is supported), but otherwise those are completely 
different.

On November 17, 2020 10:31:12 AM EST, "Ludovic Courtès"  wrote:
>Hi Stephen,
>
>(+Cc: guix-devel@gnu.org.  You can post without being subscribed.)
>
>Stephen Christie  skribis:
>
>> I have done a lot of work with the Conan package manager, a c++
>> language package manager, that has grown in capability. It is not
>> fully functional, but works on the hash of the key parameters of the
>> package (name, version, etc.) to find the right reproducible
>> binary. Two important parameters are "options" and "settings".
>>
>> Options are per-package, and generally affect none below it. You can
>> specify defaults for the options in the package, and also call for
>> specific options on dependencies in package "recipes". There are also
>> ways to define incompatibilities and substitutes. On the command
>line,
>> you can specify options with -o,--options, with no namespace needed
>> for the package you are installing, and package:option to specify for
>> other packages pulled in. I prefer this syntax to all the equal signs
>> you proposed (though I defer if this is standard throughout
>> Guix/Guile)
>>
>> Settings are more "system-wide", though being a language package
>> manager, it does not have a "system". The same settings are applied
>to
>> the whole tree during an install, and are usually things like
>> compiler, architecture, and build type. These settings are chosen
>> through a profile file, of which there is a default generated for a
>> given computer. Settings can also be set at the command line during
>> install with -s,--settings, but of course there is no namespacing.
>>
>> https://docs.conan.io/en/latest/mastering/conditional.html
>>
>> I think there is a lot of good stuff in Conan that Guix could learn
>> from. It's a lot closer in architecture than any of the traditional
>> system package managers.
>
>Thanks for sharing Conan’s perspective on these issues!
>
>The settings/options distinction looks like a useful one.  Like Pierre
>suggested, it’d be nice to have options that apply to the whole graph
>in
>addition to per-package options like I was focusing on.
>
>Conan’s approach to conflicting options may not be applicable to Guix.
>For instance, the manual above has this example:
>
>  def configure(self):
># …
>if self.settings.os == "Windows":
>   self.options["openssl"].shared = True
>
>  def requirements(self):
>  # Or add a new requirement!
>  if self.options.testing:
> self.requires("OpenSSL/2.1@memsharded/testing")
>  else:
> self.requires("openssl/1.0.2u")
>
>In Guix, instead of stating that OpenSSL 1.0.2u is required or that it
>needs to include shared libraries, you’d actually depend on a variant
>of
>OpenSSL that fulfills these constraints; by construction, you can be
>sure you have the intended OpenSSL variant (generally speaking, a Guix
>package dependency graph has zero degrees of liberty, unlike an
>apt/Spack/Conan graph.)
>
>As for the syntax… yeah, we could find a shorthand.  :-)  The verbosity
>in the examples I gave partly stems from the fact that these are
>per-package parameters, so you need to specify which package it applies
>to.  With “global” parameters, we could have, say:
>
>  guix install -P x11=false emacs
>
>meaning that the ‘x11’ parameter will be set to #f in all the packages
>that have such a parameter.
>
>Thanks,
>Ludo’.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi Stephen,

(+Cc: guix-devel@gnu.org.  You can post without being subscribed.)

Stephen Christie  skribis:

> I have done a lot of work with the Conan package manager, a c++
> language package manager, that has grown in capability. It is not
> fully functional, but works on the hash of the key parameters of the
> package (name, version, etc.) to find the right reproducible
> binary. Two important parameters are "options" and "settings".
>
> Options are per-package, and generally affect none below it. You can
> specify defaults for the options in the package, and also call for
> specific options on dependencies in package "recipes". There are also
> ways to define incompatibilities and substitutes. On the command line,
> you can specify options with -o,--options, with no namespace needed
> for the package you are installing, and package:option to specify for
> other packages pulled in. I prefer this syntax to all the equal signs
> you proposed (though I defer if this is standard throughout
> Guix/Guile)
>
> Settings are more "system-wide", though being a language package
> manager, it does not have a "system". The same settings are applied to
> the whole tree during an install, and are usually things like
> compiler, architecture, and build type. These settings are chosen
> through a profile file, of which there is a default generated for a
> given computer. Settings can also be set at the command line during
> install with -s,--settings, but of course there is no namespacing.
>
> https://docs.conan.io/en/latest/mastering/conditional.html
>
> I think there is a lot of good stuff in Conan that Guix could learn
> from. It's a lot closer in architecture than any of the traditional
> system package managers.

Thanks for sharing Conan’s perspective on these issues!

The settings/options distinction looks like a useful one.  Like Pierre
suggested, it’d be nice to have options that apply to the whole graph in
addition to per-package options like I was focusing on.

Conan’s approach to conflicting options may not be applicable to Guix.
For instance, the manual above has this example:

  def configure(self):
# …
if self.settings.os == "Windows":
   self.options["openssl"].shared = True

  def requirements(self):
  # Or add a new requirement!
  if self.options.testing:
 self.requires("OpenSSL/2.1@memsharded/testing")
  else:
 self.requires("openssl/1.0.2u")

In Guix, instead of stating that OpenSSL 1.0.2u is required or that it
needs to include shared libraries, you’d actually depend on a variant of
OpenSSL that fulfills these constraints; by construction, you can be
sure you have the intended OpenSSL variant (generally speaking, a Guix
package dependency graph has zero degrees of liberty, unlike an
apt/Spack/Conan graph.)

As for the syntax… yeah, we could find a shorthand.  :-)  The verbosity
in the examples I gave partly stems from the fact that these are
per-package parameters, so you need to specify which package it applies
to.  With “global” parameters, we could have, say:

  guix install -P x11=false emacs

meaning that the ‘x11’ parameter will be set to #f in all the packages
that have such a parameter.

Thanks,
Ludo’.

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

Danny Milosavljevic  skribis:

> For the embedded/flash rom side:

Note: it’s great if it can help reduce closure size, but that’s not the
goal.

> * Enable/disable building the documentation.  I really don't need a 40 MiB
> manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
> extra output, fair enough.

Extra output is better, yup.  :-)

> * Enable/disable obscure dependencies: Library packages that pull in 400 MiB 
> Qt
> into the closure for a thing no one (TM) uses should probably provide a 
> switch to
> disable that GUI.  Sometimes a package provides multiple GUIs for different
> toolkits, in which case one should be able to choose one toolkit and not
> build for the others.

Yes, that sounds like a good use case.

> * No, even in 2020, I won't start using AmigaFS, NFSv3 and whatever old
> protocol/format is still around and superseded by other protocols.
>
> The gexp-functionality of being able to select individual files of a package
> is already very useful for use cases an embedded developer would have
> (and that's there for a long time already).  So that's nice!
>
> For the kind of feature flags I have in mind, it usually means I don't want
> to have the feature *anywhere*--for example, if I don't want to have Qt or
> Kerberos or whatever, that's because I want to save the space and thus
> it should be able to be *globally* specified--at least per profile.

OK, so that’s similar to what Pierre was suggesting: “global”
parameters, or at least parameters that apply to all the packages that
know about it, not just to one package.

Sounds doable, but again, the challenge will be to build all the
combinations.

> I would advise against doing a grep -r -- --enable and introducing all those
> as parameters.  Rather I would check the closure of stuff and if the closure
> goes from 1200 MiB to 50 MiB, chances are a parameter would be nice there :)

Heh, sounds like a valid criterion.

I guess initially we’ll have to use them sparsely (in Guix proper) so we
can gain experience with them.

> From experience with Gentoo before I can tell you that the combinatory
> explosion is a real problem and most of the "more advanced" (toggled more
> switches :) ) combinations did not work the majority of the time.

Right, and that’s precisely what I’d like to avoid, at least for the
packages in Guix proper (authors of external channels might have a
different policy.)  So I suppose in Guix we’d have a policy of using
them sparsely, and only/primarily in leaf packages.

Thanks,
Ludo’.

Re: A plan for parameterized packages

[zimoun]

Hi,

On Mon, 16 Nov 2020 at 13:03, Pierre Neidhardt  wrote:

> See point 7. about conflicts:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00026.html

Quoting for instance the 2 examples:

For instance, use guile-2.2.4 instead of guile for all guile
libraries, or use pulseaudio everywhere, including in
dependencies that are not explicitly installed to the user
profile.

>From my understanding, the first case (guile) is now covered by the
“package rewriting” transformation (see package-input-rewriting IIUC).

The issue with the second case is below.


> and comments:
>
> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00177.html

>From my understanding of the Ludo’s patch and from the Marius’s message,
the both looks really similar. :-)


> https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00181.html

Quoting:

To be usable, we would need something to say "build bar and all
its inputs without pulseaudio, except for some given packages".

While this is OK with 1 parameter, it's quickly gets much more
complicated when packages have multiple parameters that maybe conflict
with one another.

Is the combinatorial conflict solvable in advance?  Well, from my point
of view, it cannot be via package transformation.

For example, let’s consider the Emacs packages and #41732.  Quoting [1]:

> Perhaps then,
>
> --8<---cut here---start->8---
> guix build -m manifest.scm --with-input=emacs-minimal=emacs-next \
>  --with-input=emacs=emacs-next
> --8<---cut here---end--->8---

Possibly. And then Magit uses emacs-no-x as an input, so we may
need to also add --with-input=emacs-no-x=emacs-next to the
command.

I'm just pointing out that the process is not as straightforward
as it might seem. So, it doesn't sound right to simply suggest

and [2]:

For example, the package emacs-magit drags emacs-no-x because of
emacs-libgit, why is emacs-minimal not enough here?

Well, the emacs-build-system depends (implicitly) on emacs-minimal,
only.  And the initial patch `package-with-emacs-next' was changing
this, only.  However, the package emacs-libgit is cmake-build-system
and the package emacs-no-x is an explicit dependency; which is another
story. :-)

these both examples show that it is already complex enough just to
rebuild all the Emacs packages using another Emacs VM (emacs-next or
REmacs, etc.).

To be clear, I am not convinced that the current package recipes are
functional enough to be able to keep under control the combinatorial
conflict; in the general case.

1: 
2: 

All the best,
simon

Re: A plan for parameterized packages

[Ludovic Courtès]

Hi,

zimoun  skribis:

>> To me the requirements for package parameters are:
>>
>>   1. it must be possible to discover them and choose them from the UI;
>>
>>   2. they must contain on-line internationalized documentation such that
>>  the UI can list a package’s parameters and their type;
>
> Except ’boolean’, which kind of type do you have in mind?  Aside that
> you did not find examples of packages requiring parameters. ;-)

Another example would be enumerated types.

>>   3. the chosen parameters when installing a package in a profile must
>>  be preserved;
>
> You mean track the parameters with ’properties’ in /manifest,
> right?

Yup!

>>   4. it must be possible to enumerate all the possible values of a
>>  parameter, and thus to build the Cartesian product of all the
>>  possible parameter combinations of a package (or of a package
>>  graph!), so we can test those combinations as much as possible.
>
> The values of the option are therefore known at package time, right?
> However, this implies restricted possibility for the type, right?

The  record has to specify a type, and the type must
have a finite universe.  So there cannot be an “integer” type, for
instance, but there can be “integer between 10 and 42”.

>> +(define (evaluate-parameter-specs specs proc)
>> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
>> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
>> +to return the replacement package.  Raise an error if an element of SPECS 
>> uses
>> +invalid syntax, or if a package it refers to could not be found."
>> +  (map (lambda (spec)
>> + (match (string-tokenize spec %not-equal)
>> +   ((spec name value)
>> +(define (replace old)
>> +  (proc old name value))
>> +
>> +(cons spec replace))
>> +   (_
>> +(raise
>> + (formatted-message
>> +  (G_ "invalid package parameter specification: ~s")
>> +  spec)
>> +   specs))
>
> Here ’proc’ could be anything, right?  But then…
>
>> +(define (transform-package-parameters replacement-specs)
>> +  "Return a procedure that, when passed a package, replaces its direct
>> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
>> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
>> +'guile-next' from the latest commit on its 'stable-3.0' branch."
>> +  (define (replace old name value)
>> +(set-package-parameter-value old name value))
>> +
>> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
>> + replace))
>> + (rewrite  (package-input-rewriting/spec replacements)))
>> +(lambda (obj)
>> +  (if (package? obj)
>> +  (rewrite obj)
>> +  obj
>
> … it is ’set-package-parameter-value’.  It is not clear in my mind.

Yes I could have used ‘set-package-parameter-value’ directly instead of
adding this ‘proc’ parameter; it would have been more readable I guess!

Thanks,
Ludo’.

Re: A plan for parameterized packages

[Ryan Prior]

On November 15, 2020, raingloom  wrote:
> Alpine already achieves an incredibly tiny install size by splitting
> packages into many outputs. We could and should do the same.
> As far as I know, they do not have parameterized packages.

I definitely support more package-splitting and dependency tree-shaking
where possible. Getting a core Guix System down to Alpine size would be
fantastic.

For example, `docker size python` shows a 171.4mb bundle size just for
Python alone, while `docker images python:alpine` shows a 44.3mb system
image, which includes a bunch of system stuff in addition to Python. It
would be great to have tools to understand what's in our big Guix
packages and how we could ship light bundles that get the job done.

Re: A plan for parameterized packages

[raingloom]

On Sun, 15 Nov 2020 21:46:58 +0100
Danny Milosavljevic  wrote:

> Hi Ludo,
> 
> nice feature!
> 
> On Sun, 15 Nov 2020 17:33:28 +0100
> Ludovic Courtès  wrote:
> 
> > An important question: do we have examples of packages for which
> > we’d like to have parameters?
> 
> For the embedded/flash rom side:
> 
> * Enable/disable building the documentation.  I really don't need a
> 40 MiB manual stored onto a 16 MiB firmware flash chip.  If that's
> better done as an extra output, fair enough.
> ..

Alpine already achieves an incredibly tiny install size by splitting
packages into many outputs. We could and should do the same.
As far as I know, they do not have parameterized packages.

Re: A plan for parameterized packages

[zimoun]

Dear,

On Sun, 15 Nov 2020 at 21:46, Danny Milosavljevic  
wrote:

> * Enable/disable building the documentation.  I really don't need a 40 MiB
> manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
> extra output, fair enough.

Related (I hope) is: build packages with several outputs.  For instance,
’git’ has several 'outputs' ("send-email", "svn", etc.), so the list of
"inputs" provides e.g., "subversion" even if I am only interested by
e.g., "git:send-email".  This matters about closure.

And it is maybe an occasion to revisit the museum, i.e., the TODO file:

** extend ‘propagated-build-inputs’ with support for multiple outputs

#+BEGIN_SRC scheme
  (outputs '("out" "include"))
  (propagated-build-inputs
`"i1" ,p1 "o1")
("i2" ,p2))
   => "include")
  ("i3" ,p3)))
#+END_SRC

For one reference:

  


All the best,
simon

Re: A plan for parameterized packages

[Danny Milosavljevic]

Hi Ludo,

nice feature!

On Sun, 15 Nov 2020 17:33:28 +0100
Ludovic Courtès  wrote:

> An important question: do we have examples of packages for which we’d
> like to have parameters?

For the embedded/flash rom side:

* Enable/disable building the documentation.  I really don't need a 40 MiB
manual stored onto a 16 MiB firmware flash chip.  If that's better done as an
extra output, fair enough.

* Enable/disable obscure dependencies: Library packages that pull in 400 MiB Qt
into the closure for a thing no one (TM) uses should probably provide a switch 
to
disable that GUI.  Sometimes a package provides multiple GUIs for different
toolkits, in which case one should be able to choose one toolkit and not
build for the others.

* No, even in 2020, I won't start using AmigaFS, NFSv3 and whatever old
protocol/format is still around and superseded by other protocols.

The gexp-functionality of being able to select individual files of a package
is already very useful for use cases an embedded developer would have
(and that's there for a long time already).  So that's nice!

For the kind of feature flags I have in mind, it usually means I don't want
to have the feature *anywhere*--for example, if I don't want to have Qt or
Kerberos or whatever, that's because I want to save the space and thus
it should be able to be *globally* specified--at least per profile.

It doesn't help one bit for space savings if package A doesn't pull in
Kerberos or latex or or Haskell bindings for a Raspberry Pi serial port
library (guess why that's oddly specific ;) ), but package B does.

However, sometimes using static libraries is better and there's no reason to
build the shared libraries.  But that's very much local to whatever I'm
trying to do (at least local to a profile if not package).

I would advise against doing a grep -r -- --enable and introducing all those
as parameters.  Rather I would check the closure of stuff and if the closure
goes from 1200 MiB to 50 MiB, chances are a parameter would be nice there :)

I guess the kind of flags I envision would be set at profile level.

From experience with Gentoo before I can tell you that the combinatory
explosion is a real problem and most of the "more advanced" (toggled more
switches :) ) combinations did not work the majority of the time.


pgpick5BhadGn.pgp
Description: OpenPGP digital signature

Re: A plan for parameterized packages

[Taylan Kammer]

An important question: do we have examples of packages for which we’d
like to have parameters?  I’d grepped for “inherit” and that yields a
few potential candidates, but also maybe a few potential non-candidates.
Would this be a good fit for them?


I suppose Emacs would be an obvious candidate, with its "no-x" as well 
as "no-x-toolkit" variants.



Taylan

Re: A plan for parameterized packages

[zimoun]

Hi Pierre,

On Sun, 15 Nov 2020 at 18:44, Pierre Neidhardt  wrote:

> For instance, what if we want to build "all packages without X support"?
> This means that the parameter must traverse all inputs recursively if we
> don't want to drag X indirectly.

It means: add this new ’optionally’ to all the packages supporting X to
be able to turn it off with the ’boolean’ type.  IIUC.


> If I understand your change correctly, the patch is only applying
> parameters to the given package and it's not propagated to the inputs,
> is that correct?

Because of ’package-input-rewriting/spec’, it is applied to all the
graph,  IIUC.

Cheers,
simon

Re: A plan for parameterized packages

[zimoun]

Hi Lduo,

On Sun, 15 Nov 2020 at 17:33, Ludovic Courtès  wrote:

> That said, this message is about a possible implementation of package
> parameters, so here we go.  :-)

Cool!


> To me the requirements for package parameters are:
>
>   1. it must be possible to discover them and choose them from the UI;
>
>   2. they must contain on-line internationalized documentation such that
>  the UI can list a package’s parameters and their type;

Except ’boolean’, which kind of type do you have in mind?  Aside that
you did not find examples of packages requiring parameters. ;-)

The answer leads to your point #4.


>   3. the chosen parameters when installing a package in a profile must
>  be preserved;

You mean track the parameters with ’properties’ in /manifest,
right?


>   4. it must be possible to enumerate all the possible values of a
>  parameter, and thus to build the Cartesian product of all the
>  possible parameter combinations of a package (or of a package
>  graph!), so we can test those combinations as much as possible.

The values of the option are therefore known at package time, right?
However, this implies restricted possibility for the type, right?


> Subject: [PATCH 1/4] DRAFT Add (guix parameters).
>
> DRAFT: Missing tests & doc.
>
> * guix/parameters.scm: New file.
> * Makefile.am (MODULES): Add it.
> ---

[...]

> diff --git a/guix/parameters.scm b/guix/parameters.scm
> +
> +;; Type of a package parameter.
> +(define-record-type*  parameter-type
> +  make-parameter-type
> +  parameter-type?
> +  (name  parameter-type-name)  ;debugging purposes only!
> +  (string->value parameter-type-string->value)
> +  (value->string parameter-type-value->string)
> +  (universe  parameter-type-universe))
> +
> +(define boolean
> +  ;; The Boolean parameter type.
> +  (parameter-type (name 'boolean)
> +  (universe '(#true #false))
> +  (value->string
> +   (match-lambda
> + (#f "false")
> + (#t "true")))
> +  (string->value
> +   (lambda (str)
> + (cond ((string-ci=? str "true")
> +#t)
> +   ((string-ci=? str "false")
> +#f)
> +   (else
> +(raise (condition
> +( (message "wrong 
> value"))

The types will be “hard-coded“ here, right?  Boolean being the simplest
example and imagination just needs to be released, right? :-)



> Subject: [PATCH 2/4] DRAFT transformations: Add '--with-parameter'.
>
> DRAFT: Missing tests & doc.
>
> * guix/transformations.scm (evaluate-parameter-specs)
> (transform-package-parameters): New procedures.
> (%transformations, %transformation-options): Add 'with-parameter'.
> ---
>  guix/transformations.scm | 39 +++
>  1 file changed, 39 insertions(+)
>
> diff --git a/guix/transformations.scm b/guix/transformations.scm

[...]

> +(define (evaluate-parameter-specs specs proc)
> +  "Parse SPECS, a list of strings like \"bitlbee=purple=true\", and return a
> +list of spec/procedure pairs, where (PROC PACKAGE PARAMETER VALUE) is called
> +to return the replacement package.  Raise an error if an element of SPECS 
> uses
> +invalid syntax, or if a package it refers to could not be found."
> +  (map (lambda (spec)
> + (match (string-tokenize spec %not-equal)
> +   ((spec name value)
> +(define (replace old)
> +  (proc old name value))
> +
> +(cons spec replace))
> +   (_
> +(raise
> + (formatted-message
> +  (G_ "invalid package parameter specification: ~s")
> +  spec)
> +   specs))

Here ’proc’ could be anything, right?  But then…

> +(define (transform-package-parameters replacement-specs)
> +  "Return a procedure that, when passed a package, replaces its direct
> +dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is a list of
> +strings like \"guile-next=stable-3.0\" meaning that packages are built using
> +'guile-next' from the latest commit on its 'stable-3.0' branch."
> +  (define (replace old name value)
> +(set-package-parameter-value old name value))
> +
> +  (let* ((replacements (evaluate-parameter-specs replacement-specs
> + replace))
> + (rewrite  (package-input-rewriting/spec replacements)))
> +(lambda (obj)
> +  (if (package? obj)
> +  (rewrite obj)
> +  obj

… it is ’set-package-parameter-value’.  It is not clear in my mind.
Does this constrain the hypothetical types?


Cheers,
simon

Re: A plan for parameterized packages

[Pierre Neidhardt]

Fantastic!

One of the biggest struggle we had when discussing it was figuring out
what to do about parameter propagation across dependencies.

For instance, what if we want to build "all packages without X support"?
This means that the parameter must traverse all inputs recursively if we
don't want to drag X indirectly.

If I understand your change correctly, the patch is only applying
parameters to the given package and it's not propagated to the inputs,
is that correct?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: A plan for parameterized packages

[Nicolò Balzarotti]

More in general, grepping for `--enable` or `--with` flags shows
a lot of potential packages (but also many false positives).


Nicolò Balzarotti  writes:

> Neat! Thanks for working on it :)
>
>> An important question: do we have examples of packages for which we’d
>> like to have parameters?  I’d grepped for “inherit” and that yields a
>> few potential candidates, but also maybe a few potential non-candidates.
>> Would this be a good fit for them?
>
> What about evince?  eps/ps/dvi rendering is disabled, according to this
> thread [1] for security reasons, so it makes sense to have it disabled
> by default.  But I'd like to be able to enable it easily.
>
> WDYT?
>
> Nicolò
>
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674
>
> Ludovic Courtès  writes:
>
>> Hello Guix!
>>
>> For some time we’ve discussed ways to achieve “parameterized
>> packages”—packages where one can from the command line or from Scheme
>> configure optional build-time features, similar to Gentoo “USE flags”.
>>
>> I still have mixed feeling about this feature: on one hand it can bring
>> much welcome flexibility, but on the other hand it can also lead us to
>> the wild west of untested package configurations and combinatorial
>> explosion.  It could also be argued that we achieve something similar
>> today by simply defining package variants when we have to:
>>
>>   
>> https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>>
>> That said, this message is about a possible implementation of package
>> parameters, so here we go.  :-)
>>
>> To me the requirements for package parameters are:
>>
>>   1. it must be possible to discover them and choose them from the UI;
>>
>>   2. they must contain on-line internationalized documentation such that
>>  the UI can list a package’s parameters and their type;
>>
>>   3. the chosen parameters when installing a package in a profile must
>>  be preserved;
>>
>>   4. it must be possible to enumerate all the possible values of a
>>  parameter, and thus to build the Cartesian product of all the
>>  possible parameter combinations of a package (or of a package
>>  graph!), so we can test those combinations as much as possible.
>>
>> This leads to the patches below.  The last one gives an example use for
>> Bitlbee: it adds a “libpurple” parameter that allows users to choose
>> whether or not to enable the optional libpurple dependency:
>>
>> --8<---cut here---start->8---
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true 
>> -n
>> The following derivation would be built:
>>/gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false 
>> -n
>> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? 
>> -n
>> guix build: error: wrong value
>> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true 
>> -n
>> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
>> $ ./pre-inst-env guix show bitlbee
>> name: bitlbee
>> version: 3.6
>> outputs: out
>> parameters: libpurple
>> systems: x86_64-linux i686-linux
>> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 
>> perl@5.30.2 pkg-config@0.29.2 python@3.8.2
>> location: gnu/packages/messaging.scm:243:2
>> homepage: https://www.bitlbee.org/
>> license: GPL 2+, FreeBSD
>> synopsis: IRC to instant messaging gateway  
>> description: BitlBee brings IM (instant messaging) to IRC clients, for 
>> people who have an IRC client running all
>> + the time and don't want to run an additional IM client.  BitlBee currently 
>> supports XMPP/Jabber (including Google
>> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter 
>> microblogging network (plus all other Twitter
>> + API compatible services like identi.ca and status.net).
>>
>> $ ./pre-inst-env guix install bitlbee 
>> --with-parameter=bitlbee=libpurple=true -p /tmp/test-bitlbee
>> The following package will be installed:
>>bitlbee 3.6
>>
>> The following derivation will be built:
>>/gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>>
>> [...]
>>
>> $ cat /tmp/test-bitlbee/manifest
>> ;; This file was automatically generated and is for internal use only.
>> ;; It cannot be passed to the '--manifest' option.
>>
>> (manifest
>>   (version 3)
>>   (packages
>> (("bitlbee"
>>   "3.6"
>>   "out"
>>   "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>>   (propagated-inputs ())
>>   (search-paths ())
>>   (properties
>> (transformations
>>   (with-parameter . "bitlbee=libpurple=true")))
>> --8<---cut here---end--->8---
>>
>> That’s it!
>>
>> We would need more things, like a ‘guix parameters’ command to show the
>> available parameters of a package and an 

Re: A plan for parameterized packages

[Nicolò Balzarotti]

Neat! Thanks for working on it :)

> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential non-candidates.
> Would this be a good fit for them?

What about evince?  eps/ps/dvi rendering is disabled, according to this
thread [1] for security reasons, so it makes sense to have it disabled
by default.  But I'd like to be able to enable it easily.

WDYT?

Nicolò

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=39674

Ludovic Courtès  writes:

> Hello Guix!
>
> For some time we’ve discussed ways to achieve “parameterized
> packages”—packages where one can from the command line or from Scheme
> configure optional build-time features, similar to Gentoo “USE flags”.
>
> I still have mixed feeling about this feature: on one hand it can bring
> much welcome flexibility, but on the other hand it can also lead us to
> the wild west of untested package configurations and combinatorial
> explosion.  It could also be argued that we achieve something similar
> today by simply defining package variants when we have to:
>
>   
> https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html
>
> That said, this message is about a possible implementation of package
> parameters, so here we go.  :-)
>
> To me the requirements for package parameters are:
>
>   1. it must be possible to discover them and choose them from the UI;
>
>   2. they must contain on-line internationalized documentation such that
>  the UI can list a package’s parameters and their type;
>
>   3. the chosen parameters when installing a package in a profile must
>  be preserved;
>
>   4. it must be possible to enumerate all the possible values of a
>  parameter, and thus to build the Cartesian product of all the
>  possible parameter combinations of a package (or of a package
>  graph!), so we can test those combinations as much as possible.
>
> This leads to the patches below.  The last one gives an example use for
> Bitlbee: it adds a “libpurple” parameter that allows users to choose
> whether or not to enable the optional libpurple dependency:
>
> --8<---cut here---start->8---
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=true -n
> The following derivation would be built:
>/gnu/store/mkknqgjsa93ajcl5d2krngizb11j1b0q-bitlbee-3.6.drv
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=false 
> -n
> /gnu/store/c2ckg51ffwgs6jni3l549k06w3jd3b7a-bitlbee-3.6
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libpurple=wat? -n
> guix build: error: wrong value
> $ ./pre-inst-env guix build bitlbee --with-parameter=bitlbee=libviolet=true -n
> gnu/packages/messaging.scm:283:5: error: libviolet: no such package parameter
> $ ./pre-inst-env guix show bitlbee
> name: bitlbee
> version: 3.6
> outputs: out
> parameters: libpurple
> systems: x86_64-linux i686-linux
> dependencies: check@0.12.0 glib@2.62.6 gnutls@3.6.12 libotr@4.1.1 perl@5.30.2 
> pkg-config@0.29.2 python@3.8.2
> location: gnu/packages/messaging.scm:243:2
> homepage: https://www.bitlbee.org/
> license: GPL 2+, FreeBSD
> synopsis: IRC to instant messaging gateway  
> description: BitlBee brings IM (instant messaging) to IRC clients, for people 
> who have an IRC client running all
> + the time and don't want to run an additional IM client.  BitlBee currently 
> supports XMPP/Jabber (including Google
> + Talk), MSN Messenger, Yahoo! Messenger, AIM and ICQ, and the Twitter 
> microblogging network (plus all other Twitter
> + API compatible services like identi.ca and status.net).
>
> $ ./pre-inst-env guix install bitlbee --with-parameter=bitlbee=libpurple=true 
> -p /tmp/test-bitlbee
> The following package will be installed:
>bitlbee 3.6
>
> The following derivation will be built:
>/gnu/store/clvs5521v5ybdw1z1yh97z2ky1dxm4d9-bitlbee-3.6.drv
>
> [...]
>
> $ cat /tmp/test-bitlbee/manifest
> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
>
> (manifest
>   (version 3)
>   (packages
> (("bitlbee"
>   "3.6"
>   "out"
>   "/gnu/store/d67r9k5hwfm5hkd1d3pbhg49fcc2jj4q-bitlbee-3.6"
>   (propagated-inputs ())
>   (search-paths ())
>   (properties
> (transformations
>   (with-parameter . "bitlbee=libpurple=true")))
> --8<---cut here---end--->8---
>
> That’s it!
>
> We would need more things, like a ‘guix parameters’ command to show the
> available parameters of a package and an ‘--all-parameter-values’ option
> to ‘guix build’ to build all the variants of a given package.
>
> An important question: do we have examples of packages for which we’d
> like to have parameters?  I’d grepped for “inherit” and that yields a
> few potential candidates, but also maybe a few potential