Re: Pinning package inputs using inferiors?
Hi, On sam., 22 oct. 2022 at 22:58, Felix Lechner via "Development of GNU Guix and the GNU System distribution." wrote: > While I am relatively new to functional package management, I find it > inconsistent that inputs in Guix are provided by variables. What do you mean by « variables »? > I believe the inputs should be provided by functions that deliver the > most suitable version of a package. For most packages, that is the > most recent version unless pinned. What do you mean by « inputs »? Currently, the “new” style refers to symbol as inputs which points to other package definitions. A package definition somehow defines a function to build a package. At this level, it is hard to have a precise meaning of « suitable version of a package», IMHO. At the CLI level, packages are referred by their name field and the most recent version is used by default; unless a specific version as ’@1.2.3’ is appended to the name. > A package definition would be the list of available versions rather > than just one version. What do you build? Do you build the matrix of all the combinations? The hard task of a package manager is to provide a set of packages at their appropriated versions that works well all together. Aside some specific cases where the compatibility across version is guarantee, I miss how a mutli-version definition could work in practise. Package transformation is somehow a way to implement package definition for several versions. It allows to rewrite the function definition (package) but in the same time to keep under control the combinations. Cheers, simon
Re: Pinning package inputs using inferiors?
On Fri, Oct 21, 2022 at 10:08:10PM +0100, Phil wrote: > Thanks Simon - I've given an example below. > > zimoun writes: > > > For an example, see python-numpy and python-numpy-next in (gnu packages > > python-xyz). > > This was my original way of handling this but in what is perhaps a niche > use of Guix by my department - it ultimately doesn't scale well, for our > use-case. > > Originally the department was small enough that there was only a handful > of applications sharing one or two common in-house libraries. > > As we've scaled-up we now have the situation where 3 or 4 common > libraries are being used by say 10 applications. > > We have rapid release schedules - and want to be able to release the > common libraries on a weekly basis. But the time to sign-off on a > common library takes a few days per application, so it's not practical for > every project to bump version every week - they have other priorities. > > In an ideal world automated unit and regression testing would be > comprehensive enough that we could move aggressively each week, but at > least for now that's not practical given the complex nature of signing > off the libraries and the applications which use the libraries. > > So, ideally, what we'd like to do is just have each common library > churn-out releases every week, and have the releases available in Guix, > but without having an obligation on dependent applications to adopt > these changes if they don't want to. > > Note all libraries and applications share the same channel - one > solution would be to have each library in their own channel, but this > feels ugly to me. > > Our solution (somewhat tricky to explain without a whiteboard - > apologies!) is to co-locate the package definition of the common library > in the common library repo itself - we call it something like > .requirements.scm and it is naturally kept in lockstep with the code in > that repo that the definition will build. This is very different to > traditional Guix where channels contain definitions separately in a > different repo to the code those definitions describe how to build. > > We then have a job in our CI/CD system that allows us to give a tag on the > common library repo, and the name of an application that uses the common > library. > > The job will copy the .requirements.scm into our channel inside a > private module specific to the application that uses the common library. > > The idea is that you can have many versions of .requirements.scm private > to every application package definition that references it. > > You could even read .requirements.scm using a function that clones the > application repo on-the-fly rather than statically storing it in the > channel - we haven't gone this far yet, as it makes the system even more > complex to reason about. > > This is basically the same idea as the python-numpy-next but allows for > many versions of python-numpy to co-exist by keeping them all in private > modules so they don't clash. > > It's a cool idea and works pretty well, but requires us to augment Guix > with a set of extra tools to lift and shift these private definitions > around, which complicates our setup considerably. > > It feels like wanting to make many versions of a library available at > once isn't an unreasonable way to work at-scale. However, it also feels > like a departure from the philosophy of Guix to decentralise package > definitions and to allow for a potentially large range of versions to > co-exist in the same channel commit. > > We could try to further integrate the idea into guix by writing new guix > commands to support it - we're still working out the details ourselves, > but if it works well we'd love to present it at a future Guix Days or > similar! > > In the meantime I was wondering if anyone else had a similar use-case > for Guix and if they had tried something similar or different to handle > many versions in an automated way in the same channel commit? > > Apologies that's more than I was intending to write - but hopefully that > makes some sense! If it doesn't I can try to flesh out specific example? Apologies for not slotting in the reply inline, I'm not sure exactly where to put it. This might be a good use for package transformations. Imagine the following: ;;; Python-dep-1 (define-publid python-dep1-week1 ...) (define-publid python-dep1-week2 ...) (define-publid python-dep1-week3 ...) ;;; Python-dep-2 (define-publid python-dep2-week1 ...) (define-publid python-dep2-week2 ...) (define-publid python-dep2-week3 ...) ;;; Python package (define my-python-package-base (name "my-python-package-base") ... (inputs (list python-dep1 python-dep2))) (define-public my-python-package (inherit my-python-package-base) (name "my-python-package") (inputs (modify-inputs (package-inputs python-package-base) (replace python-dep1 python-dep1-week3) (replace python-dep2 python-dep2-week2
Re: Pinning package inputs using inferiors?
Hi, On Fri, Oct 21, 2022 at 4:51 PM Phil wrote: > > have the releases available ... > but without ... an obligation on ... applications to adopt > these changes While I am relatively new to functional package management, I find it inconsistent that inputs in Guix are provided by variables. I believe the inputs should be provided by functions that deliver the most suitable version of a package. For most packages, that is the most recent version unless pinned. A package definition would be the list of available versions rather than just one version. One day, those functions could even be combined with importers in the sense that many versions are potentially available. Substitutes would only be produced and served for versions actually used, plus the most recent version. Kind regards Felix Lechner
Re: Pinning package inputs using inferiors?
On 2022-10-21 23:08, Phil wrote: [..] In the meantime I was wondering if anyone else had a similar use-case for Guix and if they had tried something similar or different to handle many versions in an automated way in the same channel commit? To handle many versions in an automated way, I once wrote this: https://gitlab.com/methuselah-0/guix-cigmon/-/tree/master It's a schedulable script that will check for updates in certain git repos (specified branches if you want), and create inherited package versions for the new commits which it can commit and push to a guix channel repository. Probably doesn't solve your problem completely, but might be useful. Example contents of a new file python-nbdev-org-babel-rev.scm with versions of python-nbdev-org-babel, automatically added to a guix channel: (define-module (python-nbdev-org-babel-revs) #:use-module (python-extras) #:use-module (guix packages) #:use-module (guix git-download)) (define-public python-nbdev-org-babel-4f195e9 (package (inherit python-nbdev-org-babel)(properties '(("generated-by" . "cigmon")))(name "python-nbdev-org-babel-4f195e9")(source (origin (method git-fetch) (uri (git-reference (commit "4f195e915eefe5cd4deec3c6aea27e4b61233f33")(url "https://github.com/methuselah-0/nbdev-org-babel.git";)))(sha256 (base32 "0ayfxnw1s9rzs1qpqqyqwhf21xk6g8psffsqzfvvl0w5k1j88dqn")) (define-public python-nbdev-org-babel-66079a5 (package (inherit python-nbdev-org-babel)(properties '(("generated-by" . "cigmon")))(name "python-nbdev-org-babel-66079a5")(source (origin (method git-fetch) (uri (git-reference (commit "66079a59aa9ee5e4479bdce597aac0f42f7fb565")(url "https://github.com/methuselah-0/nbdev-org-babel.git";)))(sha256 (base32 "0krdpc6zbdljriw6s80g08fywy0d1nq8wi3q07v3qs0b6rfz68n4")) (define-public python-nbdev-org-babel-master (package (inherit python-nbdev-org-babel-66079a5)(name "python-nbdev-org-babel-master"))) This way you can pin some packages to have an input like mypackage-<1234567> Would that be useful? Best regards, David
Re: Pinning package inputs using inferiors?
Thanks Simon - I've given an example below. zimoun writes: > For an example, see python-numpy and python-numpy-next in (gnu packages > python-xyz). This was my original way of handling this but in what is perhaps a niche use of Guix by my department - it ultimately doesn't scale well, for our use-case. Originally the department was small enough that there was only a handful of applications sharing one or two common in-house libraries. As we've scaled-up we now have the situation where 3 or 4 common libraries are being used by say 10 applications. We have rapid release schedules - and want to be able to release the common libraries on a weekly basis. But the time to sign-off on a common library takes a few days per application, so it's not practical for every project to bump version every week - they have other priorities. In an ideal world automated unit and regression testing would be comprehensive enough that we could move aggressively each week, but at least for now that's not practical given the complex nature of signing off the libraries and the applications which use the libraries. So, ideally, what we'd like to do is just have each common library churn-out releases every week, and have the releases available in Guix, but without having an obligation on dependent applications to adopt these changes if they don't want to. Note all libraries and applications share the same channel - one solution would be to have each library in their own channel, but this feels ugly to me. Our solution (somewhat tricky to explain without a whiteboard - apologies!) is to co-locate the package definition of the common library in the common library repo itself - we call it something like .requirements.scm and it is naturally kept in lockstep with the code in that repo that the definition will build. This is very different to traditional Guix where channels contain definitions separately in a different repo to the code those definitions describe how to build. We then have a job in our CI/CD system that allows us to give a tag on the common library repo, and the name of an application that uses the common library. The job will copy the .requirements.scm into our channel inside a private module specific to the application that uses the common library. The idea is that you can have many versions of .requirements.scm private to every application package definition that references it. You could even read .requirements.scm using a function that clones the application repo on-the-fly rather than statically storing it in the channel - we haven't gone this far yet, as it makes the system even more complex to reason about. This is basically the same idea as the python-numpy-next but allows for many versions of python-numpy to co-exist by keeping them all in private modules so they don't clash. It's a cool idea and works pretty well, but requires us to augment Guix with a set of extra tools to lift and shift these private definitions around, which complicates our setup considerably. It feels like wanting to make many versions of a library available at once isn't an unreasonable way to work at-scale. However, it also feels like a departure from the philosophy of Guix to decentralise package definitions and to allow for a potentially large range of versions to co-exist in the same channel commit. We could try to further integrate the idea into guix by writing new guix commands to support it - we're still working out the details ourselves, but if it works well we'd love to present it at a future Guix Days or similar! In the meantime I was wondering if anyone else had a similar use-case for Guix and if they had tried something similar or different to handle many versions in an automated way in the same channel commit? Apologies that's more than I was intending to write - but hopefully that makes some sense! If it doesn't I can try to flesh out specific example?
Re: Pinning package inputs using inferiors?
Hi Phil, On Thu, 20 Oct 2022 at 22:37, Phil wrote: > A change in a package ("dependency" in the below example) in a channel I > own has caused a conflict in another package in the same channel that depends > on it ("test-package" in the below). Whilst fixing the "test-package" > package is the right solution, this is too complicated in do in the > short-term. I need to pin "dependency" to v1.0 in test-pacakge's > propagated-inputs. Simultaneously, other packages need the new update to > the "dependency" package to use this functionality to deliver new > functionality that can't wait. > > This isn't a one-off situation; this happens frequently and I'm > interested in how other Guixers resolve this with as little friction to > users as possible? Well, I answer to this question and I do not answer to your question about inferior… > One brainwave I had was to use inferiors - but this doesn't seem to > work. Continuing from the above example we could define access to a > historical v1.0 of the dependency package for the test-package like so: > > (define dependency-inferior > ;; An inferior containing dependency v1.0. > (inferior-for-channels dependency-channels)) …instead of defining a complete inferior, why not just define 2 packages. Something as, (define-public foo (package (name "foo") (version "2.0") [...] (define-public foo-1.6 (package (inherit foo) (name "foo") (version "1.6") [...] For an example, see python-numpy and python-numpy-next in (gnu packages python-xyz). Note that most CLI as “guix install foo” will install the last version. Hence the ’-next’ trick in the package name. :-) Depending what you would like to be the “default“. Cheers, simon
Pinning package inputs using inferiors?
Hi all, A change in a package ("dependency" in the below example) in a channel I own has caused a conflict in another package in the same channel that depends on it ("test-package" in the below). Whilst fixing the "test-package" package is the right solution, this is too complicated in do in the short-term. I need to pin "dependency" to v1.0 in test-pacakge's propagated-inputs. Simultaneously, other packages need the new update to the "dependency" package to use this functionality to deliver new functionality that can't wait. This isn't a one-off situation; this happens frequently and I'm interested in how other Guixers resolve this with as little friction to users as possible? One brainwave I had was to use inferiors - but this doesn't seem to work. Continuing from the above example we could define access to a historical v1.0 of the dependency package for the test-package like so: (define dependency-inferior ;; An inferior containing dependency v1.0. (inferior-for-channels dependency-channels)) If we do this then I can get the below manifest to work, just like the example in the manual: (packages->manifest (list (specification->package "python") ;; useful for testing ;; Remove current dependency add old dependency (package/inherit test-package (propagated-inputs (modify-inputs (package-propagated-inputs test-package) (delete "dependency" (car (lookup-inferior-packages dependency-inferior "dependency" "1.0" But this isn't a practical approach - knocking "dependency" out of test-package's inputs means the unit tests would fail, so I would also have to delete the check phase - I don't want to do this, it's too much compromise. Instead I was hoping to replace the dependency *inside* the package definition with an inferior like this, so it's still available whilst the inherited package is being built. (packages->manifest (list (specification->package "python") ;; useful for testing ;; Remove current dependency, add old dependency (package/inherit test-package (propagated-inputs (modify-inputs (package-propagated-inputs test-package) (replace "dependency" (car (lookup-inferior-packages dependency-inferior "dependency" "1.0" When I try this, depending what operation I perform on the manifest, I normally get some type mismatch telling me I've used a package-inferior when Guile was expecting a package. Nothing works, alas. Thus I'm assuming the two types are not completely substitutable, and this won't work? Given this, the workaround I am employing is to replace a single package definition of "dependency" with locally scoped definitions for each package that uses it. This is duplication feels suboptimal. FWIW, a much more involved solution is to store the dependency package inside test-package's project repo (rather than my channel), and then automate copying this into the channel at build time. This is a cool exercise, but means we are decentralizing the channel's package definitions and co-locating them across many repos - which feels not in the spirit of Guix, and more like how requirements.txt files are co-located in (non-Guix) python projects. My questions are: 1. Is my above use of inferiors always doomed, or something we code make work with changes to Guix core? 2. Is there another way of handling the situation elegantly without using inferiors or duplicating package definitions at module scope. Any advice or comments gladly received! Cheers, Phil.