Re: Some concerns on the current situation on Go packaging
Hi Gabriel, Gabriel Arazas writes: > Hello! I'm a bit concerned about the current situation of packaging of > Go applications in Guix. > > Go modules also use semantic versioning [1] similarly to Rust > packages. In Guix, some of the Rust packages are packaged with > different versions [2]. This is nice (and tedious) as applications are > more likely to work as intended. > > However, there doesn't seem to be documented practice of packaging > different versions for Go applications. The lack of such section from > the manual also gives an additional impression unlike Rust packaging > [2]. I don't know much examples that can be found committed in Guix > repo but this makes adding Go applications to be more of a pain. I > mean packaging Rust apps is also a pain as much as packaging Go apps > (at least in my experience) but at least there are some explicit > guidelines you can follow which is also present when exploring the > code. > > For example, I'm about to package gum [3] and it needs a different > version of termenv (some commit after 0.11.0) compared to the packaged > version in Guix (0.8.1). I'm very tempted to add a different package > for it with a different version (which is thankfully easy to add). What I would do here is update termenv to latest, and try to use it with your new gum packaging. Chances are it'll work. You'll want to rebuild the other packages affected by the update, which you can get a list via `./pre-inst-env guix refresh -l termenv`. I think since this package is at 0.11.0, it hasn't yet reached a stable release (1.x.x). > In the long term, I'm more concerned about adding and maintaining of > these applications. I thought I should point them out and hopefully > get the community to reach to a consensus for Go packaging. Or is > there already some initiatives (or a consensus or even some > discussions) that I missed? We favor using the latest version of everything, as long as there is a strong reason not to (e.g., a test suite failing or the software not compiling). This means less packages definitions, which should be easier to maintain. For Go, I think it'd make sense to carry the latest version of a package as well as any previous *major* version that may be *required* by other packages. I hope this helps, Thanks, Maxim
Some concerns on the current situation on Go packaging
Hello! I'm a bit concerned about the current situation of packaging of Go applications in Guix. Go modules also use semantic versioning [1] similarly to Rust packages. In Guix, some of the Rust packages are packaged with different versions [2]. This is nice (and tedious) as applications are more likely to work as intended. However, there doesn't seem to be documented practice of packaging different versions for Go applications. The lack of such section from the manual also gives an additional impression unlike Rust packaging [2]. I don't know much examples that can be found committed in Guix repo but this makes adding Go applications to be more of a pain. I mean packaging Rust apps is also a pain as much as packaging Go apps (at least in my experience) but at least there are some explicit guidelines you can follow which is also present when exploring the code. For example, I'm about to package gum [3] and it needs a different version of termenv (some commit after 0.11.0) compared to the packaged version in Guix (0.8.1). I'm very tempted to add a different package for it with a different version (which is thankfully easy to add). In the long term, I'm more concerned about adding and maintaining of these applications. I thought I should point them out and hopefully get the community to reach to a consensus for Go packaging. Or is there already some initiatives (or a consensus or even some discussions) that I missed? [1]: https://go.dev/doc/modules/version-numbers [2]: https://guix.gnu.org/manual/devel/en/html_node/Rust-Crates.html#Rust-Crates [3]: https://github.com/charmbracelet/gum
Re: Go packaging
Ludovic Courtès transcribed 0.4K bytes: > Leo Famulari skribis: > > > I think we should do "go-$upstreamname" instead. Golang is not the name > > of the language, but rather the domain name (go.org was apparently not > > available), and a term that has been adopted by the community. But, it > > would be good to save 4 characters here. > > I’m all for “go-” instead of “golang-”. > > Ludo’, happy to help paint the shed. :-) Yes, go for it. My 2go's (and good to see some progress in Go on Guix). -- ng0 GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588 GnuPG: https://dist.ng0.infotropique.org/dist/keys/ https://www.infotropique.org https://ng0.infotropique.org signature.asc Description: PGP signature
Re: Go packaging
On Wed, Oct 04, 2017 at 10:22:25AM -0400, Leo Famulari wrote: > That package.json file is not a standard thing in the Go world. > I've found that Go applications use a variety of dependency manifest > formats, or just use Git submodules. Guix is a good thing then :). Also it means that they don't really enforce a dependency graph. How can you enforce something if you have many implementations? The enforcement is only at the single package level. > Rather, I think we should have a special go-package procedure, used in > the inputs field of the calling application, which would build the > relevant library modules of the correct Git commit. Does that make > sense? Yes. Since you can do a 'go build' I think it is possible to do this in a traditional way. It sucks that GO has so many small dependencies (similar to the node mess). But maybe we can import them somehow. Does the build tool show the graph? It is interesting that different packages have different git checkout dependencies (so different hash values for the same package go-ssl or whatever). For developers this is great because users end up with the exact same dependency graph. But for Guix I think we can ignore this. It is what we are doing today. It is easy to create a deployment environment in Guix that was never tried before. Therefore, we also don't really care. We just provide the latest and see if that works. So, I suggest to import just one version of go-ssl and cross fingers it works. If it doesn't - well then it gets a bit harder and we'll have to deal with multiple versions. > > We ought to have a look at how Nix packaged Go builds because they are > > already have a solution. Be interesting to see if they found a way to > > compile packages 'greedily', the way Python does it. > > I looked at their build system itself a few weeks ago when I was still > learning how Go compilation works. I agree, it would be fruitful to see > how they handle the issues I've raised here. Any update? Pj. --
Re: Go packaging
Leo Famulari skribis: > I think we should do "go-$upstreamname" instead. Golang is not the name > of the language, but rather the domain name (go.org was apparently not > available), and a term that has been adopted by the community. But, it > would be good to save 4 characters here. I’m all for “go-” instead of “golang-”. Ludo’, happy to help paint the shed. :-)
Re: Go packaging
On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote: > Based on my work creating a go-build-system and packaging a non-trivial > Go application [0], I want to start a discussion on how we can > efficiently package Go software in Guix. Another question, which is bikesheddy, is how to name Go packages in Guix. So far, I've used "golang-$upstreamname", which is how Petter named the work-in-progress packages I am finishing. This leads to very long package names, but I don't know a better way to ensure unique names for each package. I think we should do "go-$upstreamname" instead. Golang is not the name of the language, but rather the domain name (go.org was apparently not available), and a term that has been adopted by the community. But, it would be good to save 4 characters here. signature.asc Description: PGP signature
Re: Go packaging
On Wed, Oct 04, 2017 at 06:19:18AM +0200, Pjotr Prins wrote: Thanks for your comments, Pjotr! > Thanks Leo for the explanation. Now I understand why Go programs, such > as the IPFS implementation, have so many dependencies... Yes, so many. As for transitive dependencies... well, I probably won't package IPFS in my free time because of the huge number of packages required. > What I understand now is that packages get built 'lazily' and there is > really no way to force a build - other than running the target > software. Not exactly — you can build a Go library module with `go install name-of-module`. My packaging of Syncthing's dependencies does this, creating '.a' static library archives, which are used later when building Syncthing itself. https://github.com/lfam/guix/blob/contrib-syncthing/gnu/packages/syncthing.scm The difficulty for Guix is that we can't build the entire library (all its modules) with a single command. As far as I can tell, we'd have to recurse the filesystem tree of the library source code and try building each directory. It sounds unreliable to me. > I noticed the hash values in dependencies, so these are git checkouts > (i.e., they are versioned, but not in the standard sense). Right, there are useful "version" references in the form of the Git commit hashes, but the problem is that it seems each Go application uses a different commit. > When I read the package.json file it includes things like: > > "gxDependencies": [ > { > "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52", > "name": "go-log", > "version": "1.2.0" > }, > { > "author": "whyrusleeping", > "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M", > "name": "go-libp2p-secio", > "version": "1.1.8" > }, > { > "author": "whyrusleeping", > "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo", > "name": "go-libp2p-crypto", > "version": "1.5.0" > }, > pac > > indeed, there are even two versions in there for 'whyrusleeping' > deep dependencies ;). (would that be a reference to Ruby's Why?). In this case, I don't know what whyrusleeping refers to. go-libp2p-secio and go-libp2p-crypto are the names of the programs themselves. > On the surface, similar to Rubygems, I think it is no problem to > distribute source packages from Guix and have them compile on the fly. > In a way that is also an interesting model for late optimizations - > something we are lacking in our current infrastructure. What I think > we should do is import above json file and generate GNU binary > packages that are GO source bundles. That package.json file is not a standard thing in the Go world. I've found that Go applications use a variety of dependency manifest formats, or just use Git submodules. > Provided GO can use a live build directory outside the store it will > only compile bundles once, on demand. Ruby 'compiles' or interprets > every time, so that is one up on Go ;). That target directory would be > mutable, so that is a downside, a potential security risk. I considered something like this earlier in this effort, but it's not what I'm proposing now. As you say, it's messy to compile outside of the store at run-time, and I don't think we need to do it. Rather, I think we should have a special go-package procedure, used in the inputs field of the calling application, which would build the relevant library modules of the correct Git commit. Does that make sense? > We ought to have a look at how Nix packaged Go builds because they are > already have a solution. Be interesting to see if they found a way to > compile packages 'greedily', the way Python does it. I looked at their build system itself a few weeks ago when I was still learning how Go compilation works. I agree, it would be fruitful to see how they handle the issues I've raised here. signature.asc Description: PGP signature
Re: Go packaging
Thanks Leo for the explanation. Now I understand why Go programs, such as the IPFS implementation, have so many dependencies... What I understand now is that packages get built 'lazily' and there is really no way to force a build - other than running the target software. I noticed the hash values in dependencies, so these are git checkouts (i.e., they are versioned, but not in the standard sense). It is an interesting approach - because it does guarantee people are using the same (deep) software stack. I noticed that too: On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote: > Based on my work creating a go-build-system and packaging a non-trivial > Go application [0], I want to start a discussion on how we can > efficiently package Go software in Guix. > > Go software is developed rather differently from most of what we > package, and I think our package abstraction does not fit Go libraries > well. > > The primary differences are 0) Go libraries are typically bundled as > source code, 1) Go software is typically unversioned, 2) static archives > (.a) are the norm, and 3) Go libraries are expected to be built as part > of the build process of the calling application. That is, there is no > standard way to build an entire Go library on its own. > > Elaboration: > > 0,1) The problem with every application bundling unversioned libraries > is that we need to package a different Git commit of the library for > every application we package, or risk our packages not working. Adding > package variants per-version is a bit messy currently. > > 3) As an example of how Go libaries are built piecemeal, take the core > networking library, 'golang.org/x/net' [1]. It includes dozens of > submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build > all these submodules with a single command. Instead, each one is built > when it is needed during the build process of the calling application. > There are no build scripts. The library compilation process is > standardized as, for example, `go install golang.org/x/net/ipv4`. > > This means that the entire networking library would consist of several > dozen Guix packages, multiplied by the number of different Git commits > required by the calling applications. It's unreasonable, in my opinion. > > My suggestion is that we have two layers of Go library packages: a) a > template layer that includes the source URI, unpack-path, and other > package metadata, and b) a layer to create instances of the package > within the inputs field of the calling package. > > Perhaps the instantiation layer could look like this in practice: > > (define-public my-go-program > [...] > (inputs >`(("golang.org/x/net" > ,(go-package golang-org-x-net > (version "ffcf1bedda") > (import-paths '("golang.org/x/net/ipv4" > "golang.org/x/net/context" > "golang.org/x/net/ipv6" > "golang.org/x/net/internal/iana"))) When I read the package.json file it includes things like: "gxDependencies": [ { "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52", "name": "go-log", "version": "1.2.0" }, { "author": "whyrusleeping", "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M", "name": "go-libp2p-secio", "version": "1.1.8" }, { "author": "whyrusleeping", "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo", "name": "go-libp2p-crypto", "version": "1.5.0" }, pac indeed, there are even two versions in there for 'whyrusleeping' deep dependencies ;). (would that be a reference to Ruby's Why?). On the surface, similar to Rubygems, I think it is no problem to distribute source packages from Guix and have them compile on the fly. In a way that is also an interesting model for late optimizations - something we are lacking in our current infrastructure. What I think we should do is import above json file and generate GNU binary packages that are GO source bundles. Provided GO can use a live build directory outside the store it will only compile bundles once, on demand. Ruby 'compiles' or interprets every time, so that is one up on Go ;). That target directory would be mutable, so that is a downside, a potential security risk. What you are saying is that, inside the build system, we pull in all packages as sources and do a complete compile - which flies in the face of how dependencies are built independently today. One question is, can you force that full compilation? We ought to have a look at how Nix packaged Go builds because they are already have a solution. Be interesting to see if they found a way to compile packages 'greedily', the way Python does it. Pj.
Go packaging
Based on my work creating a go-build-system and packaging a non-trivial Go application [0], I want to start a discussion on how we can efficiently package Go software in Guix. Go software is developed rather differently from most of what we package, and I think our package abstraction does not fit Go libraries well. The primary differences are 0) Go libraries are typically bundled as source code, 1) Go software is typically unversioned, 2) static archives (.a) are the norm, and 3) Go libraries are expected to be built as part of the build process of the calling application. That is, there is no standard way to build an entire Go library on its own. Elaboration: 0,1) The problem with every application bundling unversioned libraries is that we need to package a different Git commit of the library for every application we package, or risk our packages not working. Adding package variants per-version is a bit messy currently. 3) As an example of how Go libaries are built piecemeal, take the core networking library, 'golang.org/x/net' [1]. It includes dozens of submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build all these submodules with a single command. Instead, each one is built when it is needed during the build process of the calling application. There are no build scripts. The library compilation process is standardized as, for example, `go install golang.org/x/net/ipv4`. This means that the entire networking library would consist of several dozen Guix packages, multiplied by the number of different Git commits required by the calling applications. It's unreasonable, in my opinion. My suggestion is that we have two layers of Go library packages: a) a template layer that includes the source URI, unpack-path, and other package metadata, and b) a layer to create instances of the package within the inputs field of the calling package. Perhaps the instantiation layer could look like this in practice: (define-public my-go-program [...] (inputs `(("golang.org/x/net" ,(go-package golang-org-x-net (version "ffcf1bedda") (import-paths '("golang.org/x/net/ipv4" "golang.org/x/net/context" "golang.org/x/net/ipv6" "golang.org/x/net/internal/iana"))) Or something like that. What do you think? [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28586#26 [1] https://godoc.org/golang.org/x/net signature.asc Description: PGP signature