Re: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags)

[François]

Hello,

After a hiatus I am trying to package several softwares written on Golang
(I would like to have terraform and go-jsonnet for example) and I have
some problems with the current implementation so I am resurrecting this
old mail from my Draft folder.

Looking at it I think it is mostly reformulations of what Katherine and
Sarah already wrote on the thread but several months later it could be
a useful reminder.

I have not really the energy at the moment to work on this alone but I
would be interested for team work.

On Thu, Sep 30, 2021 at 10:31:08AM -0500, Katherine Cox-Buday wrote:
> Sarah Morgensen  writes:
> 
> >> IMO, module resolution and graph dependencies in Go are complicated enough
> >> that I'd much rather take the effort we put in trying to replicate and keep
> >> up with the Go toolchain's behavior, and spend that effort elsewhere.
> >>
> >> Does anyone have opinions on this?
> >
> > Hmmm.  Setting aside whether or not we want to use external tools in
> > general...
> >
> > If we use the Go tool, we shift the problem domain into "translating between
> > `go' and Guix."

I felt uneasy when we reimplemented some inherently Go-specific pieces
of software like go.mod file parsing and dependency resolution. It seems
so brittle. So I think that for importer specifically we should be able
to use external programs like the Go toolchain to be in $PATH. And,
given the Go tool would have proper interfaces, we would indeed just
have to do some sort of translation.

But as to the exact "how" it is not easy.

Ideally we could split the tasks in severals steps.
1. identify all dependencies on form of go modules paths
2. identify source repositories (generally identified by a git-ref) for each of 
those modules
3. download content of source repository
4. populate a ready-to-use source code local cache
5. build

As I see it in Guix we want :
- 1 and 2 in the importer;
- 3 is the "origin" method;
- 4 and 5 in the build system.

`go mod download` does all of 1, 2 and 3. The translations problems are
how to extract data from this output to isolate dependency management on
the importer and having the necessary info for the "origin" method. Nix
bypass completely the problem by using the complete go mod cache as its
"origin" (or more recently "vendored" dependencies see this Nix change
to go mod vendor[4]) for each Go package. This solution is not viable
for us as we want to be able to reuse Go modules.

For the build system part I think we want to use the GOMODCACHE format as
output of Guix package and to be able to do union-dir of all dependencies
when we want to build a leaf package (e.g. using GOPROXY=file:/// or
something like that).

> > For example: when Go downloads a module, it only saves the module, not the
> > whole repository*, so we can't use that to generate the hash.  (* Except it
> > does if you used GOPROXY=direct, but that's in the module cache, and only a
> > bare repository.)
>
> We could use the ~GOMODCACHE~ environment variable[0] to specify where these 
> files are placed.

Yes we want to do that but it leaves open as to how we create this
directory structure. I think it is good to use Go tooling at "guix import"
time but not at build time. In consequence the source must be downloaded
either from upstream git repo (bare or not I do not know) or as zips
from proxy.golang.org. The zip solution could just parse the directory
contents of GOMODCACHE as described in [5]. I would rather import git
repositories closer to real source but it seems more difficult to do.

Even with zip, it leaves open the problem of mapping source zips with
Guix packages and to handle the dependencies correctly.

Any interest on the matter ?

François

[4]: 
https://github.com/NixOS/nixpkgs/pull/86376/commits/9761128d2da7bf4d878982918242e43ae28f9b94

[5]: https://github.com/golang/go/issues/35922#issuecomment-560464243

cd go/src/golang.org/x/tools/gopls   # or any other module
export GOPATH=$(mktemp -d)
go mod download
find $GOPATH/pkg/mod/cache/download -type f | \
grep '\.\(mod\|info\|zip\)$' | \
sed -e "s,$GOPATH/pkg/mod/cache/download,https://proxy.golang.org,;

Re: Freenode Administration

[François]

Hello,

On Wed, May 19, 2021 at 01:54:45PM -0400, Bone Baboon wrote:
> Will Guix continue to use Freenode?
> 
> Will Guix create #guix channels on other IRC networks? (Several of the
> resignation letters refer to )

The channel #guix will probably continue to exist on Freenode but most
users are switching to libera.chat.

As we talk the logs on http://logs.guix.gnu.org/guix are now those of
#guix on libera.chat

I have the feeling that the Guix website will be updated soon to reflect
this move.

> Will Guix start it's own IRC server(s)?

It is not an option discussed and I think it would be a misplacement of
Guix scarce resources.

> # IRC alternatives / compliments

On alternatives you missed Matrix[1] which is federated (like email)
and have several clients includig weechat for text-based interface.

Several people (and I am one of them) already uses Matrix clients to
access IRC networks with bridges (which libera.chat does not already
have but work is in progress).

There is even some ongoing work ([2], [3]) on making Matrix more
peer-to-peer but it is still completely experimental.


[1]: https://matrix.org
[2]: https://matrix.org/blog/2021/05/06/introducing-the-pinecone-overlay-network
[3]: https://matrix.org/blog/2020/06/02/introducing-p-2-p-matrix

Re: [bug#48173] [PATCH] maint: Do not xz-compress ISO images.

[François]

On Mon, May 03, 2021 at 02:15:22AM +0200, Julien Lepiller wrote:
> LGTM, but what should we do about the strings?

sed -i 's/\.iso\.xz/.iso/g' po/doc/guix-manual.*.po

and add that to the patch ?

(notwithstanding some misunderstanding related to how translation works
as I know nothing about that)

Re: go-build-system possible improvments

[François]

Hello Maxim,

On Tue, Apr 20, 2021 at 10:00:15PM -0400, Maxim Cournoyer wrote:
> That's exactly what we're doing now (disable Go module with
> GO11MODULE=off and use GOPATH to find the sources).  Debian does
> something lazy like just calling 'go module vendor', which builds a huge
> directory filled with all the sources needed.  So does Nix.  Only Gentoo
> goes to some length, from what I had seen last I check.

Do you have some links to share on how Gentoo does it?

go-build-system possible improvments

[François]

Hello,

I come back to public guix-devel as I think it can be of interest to
others too.

On Sat, Apr 17, 2021 at 08:57:03PM -0400, Maxim Cournoyer wrote:
> JOULAUD François writes:
> 
> > I am still unsure of what to do with versions and Go but the ability to
> > pin version will surely be useful in some way.
> 
> Yeah, I was expecting it might unlock building the new protobuf package
> especially a couple version backs, it had a really messy dependency
> chain including past versions of itself, but it turns out that for now
> the bigger problem is the lack of support for Go modules.

Yes. This is the biggest lock. Supporting go modules will really unlock
a lot of things.

> I'm glad to hear it!  I had read lots about Go and had some kind of plan
> for the GOPROXY (the project doing best in this regard is Gentoo); in
> theory it's possible to populate a cache with the source artifacts in a
> given directory (probably via a union of the individual packages
> contribution to the cache), then setup GOPROXY as a file server to
> provide these at build time.

I just made a lot of reading this week-end and I was more oriented towards
using GOPROXY=off and populating on-disk [module cache] which have the
exact same organisation as GOPROXY as it is indeed the last layer of
proxying. If it works it will avoid the need for a specific local server.

Concerning the former dependeency loop with google-protobuf it was
(before 1.26) indeed necessary to have all go.mod files for all versions
referenced recursively. Go downloads (or need to have in cache) only
the go.mod files though because once it have the go.mod the [Minimal
Version Selection] kicks in and keep only the higher version
requested[^1].

So we could cheat by putting only go.mod files for older versions in
out local cache without the need for anything else and it should work.

The same thing applies to conditional compilation. Go needs the go.mod
files but don't need the code itself until it tries to compiles it.

So we have probably a safe way to import those packages without too
many useless packages in our dependency graph even if it can require a
solution to easily add specific go.mod files in our outputs.

They, at Go, are nevertheless [working][lazy-loading] to modify this
behaviour in order to avoid useless network calls to GOPROXY but it
could not land in Go 1.15. It should be there in next release.


[module cache]: https://golang.org/ref/mod#module-cache
[lazy-loading]: 
https://go.googlesource.com/proposal/+/master/design/36460-lazy-module-loading.md
[Minimal Version Selection]: https://research.swtch.com/vgo-mvs
[^1]: Which seems strange until you understand that you always require a
"minimal" version in Go. The minimal version needed to build a project
is thus the maximum of all minimal required versions. The link explains
this a lot better than me ;-)

François

P.S. just as writing this I found that if we want to limit the number
of versions of Guix-packaged Go modules we could devise a mechanism to
force upgrade (go get -u) to the latest in-Guix version. This open a
lots more question to me than it resolves.

P.P.S. other solution, perhaps simpler is to not support go.mod at all
(GO11MODULE=off) and to populate the local GOPATH as we see fit. I think
(but I am not sure) that Debian took this route. Not found of this
solution, I think we have the means to do better in Guix and to keep
more in touch with upstream methodologies.

Re: GIT_EXEC_PATH Re: 1.2.1 pre-release testing

[François]

On Sat, Apr 17, 2021 at 01:24:51PM -0400, Leo Famulari wrote:
> On Sat, Apr 17, 2021 at 01:35:45PM +0200, François wrote:
> > GIT_EXEC_PATH is very probably at fault.
> > 
> > It is set via profile to allow to use git subcommands installed by other
> > packages but it has no sane default when we are outside a profile.
> > 
> > Perhaps git should have a wrapper to manage this use-case.
> 
> I think it does work properly in Guix, but requires the user to log in
> again after installing Git. That's because a lot of Guix stuff is set up
> in the login shell initialization scripts. It might even be enough to do
> `bash --login` after installing the package.

I had exactly the same problem on foreing distro after installing git
and trying to use it without reloading ~/.guix-profile/etc/profile in
my current shell. So yes, it is just a matter of refreshing the current
environment.

> I guess that many of us install Git right away, so we don't notice a
> problem.

Probably.

GIT_EXEC_PATH Re: 1.2.1 pre-release testing

[François]

Hello,

On Sat, Apr 17, 2021 at 01:43:33AM -0400, Leo Famulari wrote:
> On Fri, Apr 16, 2021 at 10:40:55PM -0500, jcguu95 wrote:
> >2.1. Better support for using git during installation process?
> > 
> >Before `guix system init /mnt/etc/config.scm /mnt', I'd like to getch
> >config.scm from my github repo. So I ran `guix package -i git'. But
> >when I `git clone ...' as usual, there's an error "git:
> >'remote-https' is not a git command .See 'git --help'".

> I also found some suggestions regarding $GIT_EXEC_PATH needing to be
> set.

GIT_EXEC_PATH is very probably at fault.

It is set via profile to allow to use git subcommands installed by other
packages but it has no sane default when we are outside a profile.

Perhaps git should have a wrapper to manage this use-case.

François

Re: [SPITBALL] Jehanne as another kernel option / porting target

[François]

Hello,

On Fri, Mar 19, 2021 at 12:44:47PM -0400, Joshua Branson wrote:
> pinoaffe  writes:
> > raingloom writes:
> >
> >> seL4 would be cool too.


> So essentially most of the active hurd developers considered a port to a
> different microkernel to be impractical.  :(
> 
> However, one of the main hurd developers (he has since stepped away from
> active development), started a hurd clone:  x15.

On the microkernel front there is also [Genode] which seems to have
nice ideas. I don't know if it would suit Guix or not.

[Genode]: https://genode.org/

Re: Proposal to remove the off-topic, not free software related thoughtcrime accusations from the Guix project pages on GNU.ORG websitew

[Alexandre François Garreau]

Le dimanche 13 octobre 2019, 15:05:06 CEST Kete via Discussions about the 
development of the GNU system a écrit :
> Besides, there is no way these men are getting these transactions
> without assault or coercion taking place. Just because Epstein mediated
> that force so that the clients didn't have to get their hands dirty is
> no reason to make a stink over how preying on defenseless people in a
> peaceful manner is not violence. There is no way they could do that
> without someone doing the violence for them. The money cleans that from
> their conscience, but they are still involved with the abuse because it
> is part of the service that they buy.

What rms said is that Minsky wouldn’t have been *aware* of it.  And there’s a 
difference between doing, supporting, ordering, knowing, and unknowingly 
benefiting, especially legally.

Anyway, it was already said that Minsky didn’t do anything:

https://pjmedia.com/instapundit/339725/