Re: Updates for Go

[wolf]

On 2023-08-22 13:14:05 +, Attila Lendvai wrote:
> > each package for those two to work properly. Also, while having pinned
> > versions of dependencies upstream seems like the consensus, I'm not sure
> > we'd like doing that, be it for the exponential CI work that would be
> > required.
> 
> 
> not arguing either way, FWIW:
> 
> - rumour has it that golang compiles very fast, and

Usually people claiming this assume there is already a compiler cache present.
The compiler heavily caches the work done.

Sure, golang compiles faster than C++ for example, but anecdotal data point: at
$DAYJOB we had to start persisting the compiler cache to make CI fast enough.

> 
> - IIUC currently the go build system in guix does not reuse build artifacts, 
> i.e. it recompiles everything for each leaf package.
> 
> -- 
> • attila lendvai
> • PGP: 963F 5D5F 45C7 DFCD 0A39
> --
> “What you do speaks so loud I cannot hear what you say.”
>   — Ralph Waldo Emerson (1803–1882)
> 
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: Updates for Go

[Katherine Cox-Buday]

On 8/27/23 9:41 AM, wolf wrote:

Sure, golang compiles faster than C++ for example, but anecdotal data point: at
$DAYJOB we had to start persisting the compiler cache to make CI fast enough.


I've seen similar things done at companies. This is perhaps an 
interesting avenue to pursue later: if Go doesn't support 
linked-libraries by that time, maybe our "linked libraries" could be to 
store what the compiler caches. More investigation needed obviously.


But, Go does compile very fast, and that's a metric the Go team tracks 
and tries to improve.



- IIUC currently the go build system in guix does not reuse build artifacts, 
i.e. it recompiles everything for each leaf package.


That is my understanding as well. We bring in the source for any package 
that requires it.


--

Katherine

Re: Updates for Go

[Katherine Cox-Buday]

On 8/25/23 6:29 PM, John Kehayias wrote:

I've not been following in detail this discussion, but where do we currently 
stand? Is the proposed Go 1.21 patch basically ready?


As far as I know, yes. I've been using it locally since I submitted the 
patch, and things seem to be working as expected. I think the Guile code 
still needs a review, but as far as I know, it's a pretty standard package.



Should we create a branch and build job to start seeing how far we get in 
making 1.21 the default Go in Guix?


I think we should do a two-phase process:

1. Check the package in.

2. Begin working on making it default.

There is utility in having the package available for people to use prior 
to it being the default.



Like others, I have a few random Go packages (a bunch locally I really need to 
clean up too) and am not familiar with the language and our packaging much. 
Still, if I can help review/push some patches and get things moving, please let 
me know.


The following would help:

1. Testing that the new package behaves correctly for them

2. Submitting individual patches for packages they work with to manually 
specify Go v1.21.0 as the version to use. We can then remove this later.


3. Investigating what work will need to be done for our larger Go 
packages with many leaves.


4. Setting up a Go branch to do the work to cut everything over (I've 
volunteered to do this, but I don't know much about it, so if someone 
does and gets to it first that will be a pleasant surprise!).



And thanks for all your work here, it is appreciated!


You're welcome! Thanks for asking!

--

Katherine

Re: Updates for Go

[John Kehayias]

Hi Katherine,

On Wed, Aug 23, 2023 at 10:12 AM, Katherine Cox-Buday wrote:

> On 8/22/23 8:24 AM, Felix Lechner via Development of GNU Guix and the
> GNU System distribution. wrote:
>> Hi Attila,
>>
>> On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai  wrote:
>>>
>>> currently the go build system in guix does not reuse build artifacts
>>
>> Can Golang reuse build artifacts?
>
> I don't think it's recommended right now. See discussion here:
>
> - 
> - 
>
> Summary:
>
> It sounds like due to the way the compiler can optimize code (e.g.
> inlining), `buildmode=shared` is not recommended, but in the future they
> are looking at allowing linking against a single shared library.

I've not been following in detail this discussion, but where do we currently 
stand? Is the proposed Go 1.21 patch basically ready? Should we create a branch 
and build job to start seeing how far we get in making 1.21 the default Go in 
Guix?

Like others, I have a few random Go packages (a bunch locally I really need to 
clean up too) and am not familiar with the language and our packaging much. 
Still, if I can help review/push some patches and get things moving, please let 
me know.

And thanks for all your work here, it is appreciated!

John

Re: Updates for Go

[Katherine Cox-Buday]

On 8/22/23 8:24 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:

Hi Attila,

On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai  wrote:


currently the go build system in guix does not reuse build artifacts


Can Golang reuse build artifacts?


I don't think it's recommended right now. See discussion here:

- https://github.com/golang/go/issues/47788#issuecomment-1019562695
- https://github.com/golang/go/issues/47788#issuecomment-1258773167

Summary:

It sounds like due to the way the compiler can optimize code (e.g. 
inlining), `buildmode=shared` is not recommended, but in the future they 
are looking at allowing linking against a single shared library.


--
Katherine

Re: Updates for Go

[Katherine Cox-Buday]

On 8/22/23 12:06 PM, david larsson wrote:

Im not a fan of Go, but I've wanted to package some Go packages. Ive 
only managed to write 2 packages for my private channel so far, but they 
were simple. If there is a guide or so somewhere explaining how to do 
this, then maybe I could complete and contribute a package that I am 
working on (with its dependencies).


The best we have currently are:

- https://guix.gnu.org/manual/en/guix.html#index-go_002dbuild_002dsystem
- https://guix.gnu.org/manual/en/guix.html#index-go

Guix packages are always welcome! The biggest help at the moment would 
be helping to bring existing packages onto Go v1.21.0 so that we can 
make that the default, and improving the Go importer so that people like 
you can just run `guix import go ` most of the time and get a 
working package out of it.



Thank you for giving Go in Guix some attention.


You're welcome! I love Guix and appreciate the work of all the other 
maintainers. Together, we have something special :)


--
Katherine

Re: Updates for Go

[david larsson]

On 2023-08-17 16:25, Katherine Cox-Buday wrote:
[..]


Even if you dislike Go, but can work your way through a package,
please consider signing up!


Hi,

Im not a fan of Go, but I've wanted to package some Go packages. Ive 
only managed to write 2 packages for my private channel so far, but they 
were simple. If there is a guide or so somewhere explaining how to do 
this, then maybe I could complete and contribute a package that I am 
working on (with its dependencies). In particular I got stuck packaging 
nwg-menu with its dependencies, such as gotk3, which will probably take 
me a long time to finish at my current skill level. I might be able to 
contribute other packages as well in the future when Ive learned more 
about it.


Thank you for giving Go in Guix some attention.

Best regards,
David

Re: Updates for Go

[Development of GNU Guix and the GNU System distribution.]

Hi Attila,

On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai  wrote:
>
> currently the go build system in guix does not reuse build artifacts

Can Golang reuse build artifacts?

Kind regards
Felix

Re: Updates for Go

[Attila Lendvai]

> each package for those two to work properly. Also, while having pinned
> versions of dependencies upstream seems like the consensus, I'm not sure
> we'd like doing that, be it for the exponential CI work that would be
> required.


not arguing either way, FWIW:

- rumour has it that golang compiles very fast, and

- IIUC currently the go build system in guix does not reuse build artifacts, 
i.e. it recompiles everything for each leaf package.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“What you do speaks so loud I cannot hear what you say.”
— Ralph Waldo Emerson (1803–1882)

Re: Updates for Go

[Wilko Meyer]

Hi Katherine,

Katherine Cox-Buday  writes:

> Thank you for volunteering!
>
> I'm not aware of a TODO list anywhere other than the issue tracker
> (https://issues.guix.gnu.org/search?query=golang+is%3Aopen).

I've spend some time during the last days getting familiar with the
go-build-system in guix and how it works internally, and while reading
guix/build/go-build-system.scm I actually found such a list as
commentary from 2018-01-06 (e3900a4d64e):

;; TODO:
;; * Avoid copying dependencies into the build environment and / or avoid using
;; a tmpdir when creating the inputs union.
;; * Use Go modules [4]
;; * Re-use compiled packages [5]
;; * Avoid the go-inputs hack
;; * Stop needing remove-go-references (-trimpath ? )
;; * Remove module packages, only offering the full Git repos? This is
;; more idiomatic, I think, because Go downloads Git repos, not modules.
;; What are the trade-offs?

this is probably not too relevant for now, but maybe it'd be good to see
which of these bullets still apply and move those as issues to the issue
tracker (if they aren't issues yet, haven't checked this).

> Personally, I think the immediate emphasis should be on making
> bringing our Go ecosystem onto a supported version of Go (ideally
> 1.21.0). If there is consensus on that, then ensuring that some of our
> packages with larger dependency graphs compile would be a good place
> to start.

I'd definitely agree on that, bringing guixes go ecosystem to 1.21.0
should be a good and reasonable start. 

> It would also be useful to get https://issues.guix.gnu.org/65317 (add
> go-1.21) reviewed, even if you don't have commit access. I've been
> exercising the package since I sent the patch, and I think v3 is
> correct (at least functionally), but it could use more exercising and
> a review of the scheme code.

I'll try to have a look into this later on. Have to keep this mail short
as my lunchbreak's almost over; but will definitely spend some time on
this later on this day!

Best Regards,

Wilko Meyer

Re: Updates for Go

[Josselin Poiret]

Hi Felix,

Felix Lechner via "Development of GNU Guix and the GNU System
distribution."  writes:

> From my experience of packaging Gocryptfs in Debian and here, perhaps
> some reconsideration should be given to the widely unpopular idea of
> using more package functions in Guix. Ending in asterisks, they would
> allow packagers, and perhaps even automated tools, to select exactly
> the particular versions or commits specified for each prerequisite in
> go.mod.

Package functions are really nice, but they don't mesh well with CLI
usage, as well as CI: you'd need to also export a "default" version of
each package for those two to work properly.  Also, while having pinned
versions of dependencies upstream seems like the consensus, I'm not sure
we'd like doing that, be it for the exponential CI work that would be
required.

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature

Re: Updates for Go

[Katherine Cox-Buday]

On 8/21/23 11:53 AM, Felix Lechner wrote:

Hi,

On Mon, Aug 21, 2023 at 9:11 AM Katherine Cox-Buday
 wrote:

the immediate emphasis should be on making bringing
our Go ecosystem onto a supported version of Go

 From my experience of packaging Gocryptfs in Debian and here, perhaps
some reconsideration should be given to the widely unpopular idea of
using more package functions in Guix. Ending in asterisks, they would
allow packagers, and perhaps even automated tools, to select exactly
the particular versions or commits specified for each prerequisite in
go.mod.
I have questions about this, but for now, I'm going to try and keep 
bringing the conversation back to just getting our ecosystem compiling 
on a supported version of Go.

Re: Updates for Go

[Katherine Cox-Buday]

Summary: these are good things to talk about. I think we should focus on 
using the current approach to get our Go ecosystem into a supported 
state before we talk about these things.


On 8/19/23 5:31 AM, Attila Lendvai wrote:

at one point i tried to compile some large projects written in golang in a 
reproducible way, and making sure that they use the exact same versions of all 
their dependencies.

in short: there's a philosophical mismatch between how guix and the golang crowd looks at 
building go apps. guix likes to explicitly represent every dependency in the guix 
namespace. golang has its own way of gathering the dependencies (and the nixos crowd 
don't mind "vendoring" the dependencies).


Yeah, there's a lot of unpack in this space for sure. To repeat what I 
think you're saying to ensure I understand it, the way I would 
characterize it is that there are two concerns:


1. The version of module dependencies a module is built with

2. How the toolchain resolves dependencies


(1) is not a new or unique problem to Go. Distros and software have 
always had mis-aligned views on what specific versions a package should 
use. Distros want to maintain fewer packages, and developers want to 
reference specific versions of dependencies. Guix is in a a better 
position in that it's theoretically possible to carry many different 
versions of the same package without conflicts. My understanding is that 
the limiting factor now are the resources that go into doing that (e.g. 
build-farms, cognitive overhead, etc.).


I don't think (2) is actually a problem. A lot of projects like to 
"vendor" their dependencies (i.e. check in the version of their 
dependency in the same repository as the host module), and there's valid 
reasons to do that. Fortunately, Go's vendoring is strongly tied to 
specific versions and the code is checksummed to ensure the vendored 
copy isn't modified. So it follows that all Guix has to do is ignore 
vendored dependencies and refer to the exact same version of the Guix 
package.


That is to say, at least the way I've framed it, I think (1) is the root 
of any issue that exists.



i've also worked on the golang importer (https://issues.guix.gnu.org/55242 
which needs a bit more care). once it works reliably enough, then it'd be 
possible to import the entire transitive closure of the dependencies into an 
isolated namespace in guix, which would be a halfway solution between the two 
conflicting philosophies.
Very nice! Our Go importer has improved so much, and further 
improvements such as these will make managing the Go ecosystem even easier.

for now i've abandoned that endeavour in favor of adding binary packages to my 
channel, but i made some notes on the way:

https://issues.guix.gnu.org/43872

a go-build-system patch
http://issues.guix.gnu.org/50227
discussion with iskarian
https://logs.guix.gnu.org/guix/2021-08-31.log#024401


I don't think this is an issue specifically with Go. As long as I've 
been involved in software, there has been an issue with N software 
depending on M versions of the same thing.



https://logs.guix.gnu.org/guix/2021-08-31.log#180940
iskarian: If you search for my name and "go" or "golang" in the 
mailing list archives, you should find a lot of discussion
https://savannah.gnu.org/users/lfam
Here's a more recent message from me: 
https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00227.html
Ah, I see I've unknowingly repeated you! 
https://lists.gnu.org/archive/html/guix-patches/2021-08/msg01222.html
Heh, it's gratifying that someone else came to the same conclusion. 
It means I wasn't totally in the weeds


The smallest divisible unit of a Go repository that is independently 
distributable is now a Go module. Modules are what are resolvable, 
versioned, and check-summed. As a rule that may have exceptions: Guix 
packages should neither encapsulate anything larger nor smaller than 
that. Some of the messages you're referencing are right around the time 
modules were being reified into the Go ecosystem.


I think there is an opportunity and need for Guix to try and reach a 
consensus on what our primitives and approaches are for the Go ecosystem 
and write them down in the manual. I think our current approach is 
workable, but there's obviously still some confusion and maybe debate to 
be had.


Having said all of that, I think we should focus on using our current 
approach to get everything compiling on a supported version of Go. I 
think our packages and substitutes are probably carrying CVEs that have 
been fixed upstream, and, in my opinion, we need to resolve that ASAP.


WDYT?

Re: Updates for Go

[Development of GNU Guix and the GNU System distribution.]

Hi,

On Mon, Aug 21, 2023 at 9:11 AM Katherine Cox-Buday
 wrote:
>
> the immediate emphasis should be on making bringing
> our Go ecosystem onto a supported version of Go

>From my experience of packaging Gocryptfs in Debian and here, perhaps
some reconsideration should be given to the widely unpopular idea of
using more package functions in Guix. Ending in asterisks, they would
allow packagers, and perhaps even automated tools, to select exactly
the particular versions or commits specified for each prerequisite in
go.mod.

Thank you for all your efforts in making Guix better for everyone!

Kind regards
Felix

Re: Updates for Go

[Katherine Cox-Buday]

On 8/17/23 3:54 PM, Wilko Meyer wrote:

That being said, I'd actually be willing to put some time and effort
into Guixes Go ecosystem; even though I haven't been on Guix for that
long and would probably have to read through prior contributions to
golang.scm to get the gist on how the go-build-system and packaging all
things go work and to contribute something useful.

Is there a list of current TODOs somewhere? Or would one start by
bumping packages to build with a more recent/non-EoL go version and see
if that works out?


Thank you for volunteering!

I'm not aware of a TODO list anywhere other than the issue tracker 
(https://issues.guix.gnu.org/search?query=golang+is%3Aopen).


Personally, I think the immediate emphasis should be on making bringing 
our Go ecosystem onto a supported version of Go (ideally 1.21.0). If 
there is consensus on that, then ensuring that some of our packages with 
larger dependency graphs compile would be a good place to start.


We also really need a Go branch to host a lot of this work. I can start 
looking at what's needed for that.


It would also be useful to get https://issues.guix.gnu.org/65317 (add 
go-1.21) reviewed, even if you don't have commit access. I've been 
exercising the package since I sent the patch, and I think v3 is correct 
(at least functionally), but it could use more exercising and a review 
of the scheme code.


--

Katherine

Re: Updates for Go

[Attila Lendvai]

at one point i tried to compile some large projects written in golang in a 
reproducible way, and making sure that they use the exact same versions of all 
their dependencies.

in short: there's a philosophical mismatch between how guix and the golang 
crowd looks at building go apps. guix likes to explicitly represent every 
dependency in the guix namespace. golang has its own way of gathering the 
dependencies (and the nixos crowd don't mind "vendoring" the dependencies).

i've also worked on the golang importer (https://issues.guix.gnu.org/55242 
which needs a bit more care). once it works reliably enough, then it'd be 
possible to import the entire transitive closure of the dependencies into an 
isolated namespace in guix, which would be a halfway solution between the two 
conflicting philosophies.

for now i've abandoned that endeavour in favor of adding binary packages to my 
channel, but i made some notes on the way:

https://issues.guix.gnu.org/43872

a go-build-system patch
http://issues.guix.gnu.org/50227
discussion with iskarian
https://logs.guix.gnu.org/guix/2021-08-31.log#024401
https://logs.guix.gnu.org/guix/2021-08-31.log#180940
  iskarian: If you search for my name and "go" or "golang" in the mailing 
list archives, you should find a lot of discussion
https://savannah.gnu.org/users/lfam
  Here's a more recent message from me: 
https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00227.html
  Ah, I see I've unknowingly repeated you! 
https://lists.gnu.org/archive/html/guix-patches/2021-08/msg01222.html
  Heh, it's gratifying that someone else came to the same conclusion. It 
means I wasn't totally in the weeds

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The anxiety children feel at constantly being tested, their fear of failure, 
punishment, and disgrace, severely reduces their ability both to perceive and 
to remember, and drives them away from the material being studied into 
strategies for fooling teachers into thinking they know what they really don't 
know.”
— John Holt (1923–1985), 'How Children Learn' (1967)

Re: Updates for Go

[Development of GNU Guix and the GNU System distribution.]

On 2023-08-17 at 23:54+02:00, Wilko Meyer wrote:
> Is there a list of current TODOs somewhere? Or would one start
> by bumping packages to build with a more recent/non-EoL go version
> and see if that works out?

Most Go packages are quite dated by a few years,
so that's probably a good idea.

One does not simply drive by a thread without personal agenda,
so I hereby request thy mentorship on adding the senpai IRC client:
https://issues.guix.gnu.org/64222

I've been asking for eyes on #guix on Libera but mostly got ghosted )-;

Re: Updates for Go

[Wilko Meyer]

Hi,

Katherine Cox-Buday  writes:

> Even if you dislike Go, but can work your way through a package,
> please consider signing up!

I started picking up Golang for work related use recently again; have
been somewhat regularly writing it between 2015 and 2018-ish, but always
favored using something like C or Rust over Golang.

That being said, I'd actually be willing to put some time and effort
into Guixes Go ecosystem; even though I haven't been on Guix for that
long and would probably have to read through prior contributions to
golang.scm to get the gist on how the go-build-system and packaging all
things go work and to contribute something useful.

Is there a list of current TODOs somewhere? Or would one start by
bumping packages to build with a more recent/non-EoL go version and see
if that works out?

Best Regards,

Wilko Meyer

Re: Updates for Go

[Katherine Cox-Buday]

On 8/16/23 11:25 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:

Hi Katherine,

On Tue, Aug 15, 2023 at 12:59 PM Katherine Cox-Buday
 wrote:


There's also no one on Guix's Go team. I've created a patch to add
myself[1]


Your courage and initiative are inspiring. (Unfortunately, my skills
are lousy.) Anybody with an interest in Golang should please speak up!


Thank you, but I'm just another hacker trying to do her part to maintain 
software she loves. I am always grateful to the other Guix hackers doing 
amazing work!


I tried to emphasize in my message that even if you dislike Go, there 
are some critical packages like Docker that are essentially going 
unmaintained because they are written in Go. *This includes security 
fixes*. As it stands, I cannot recommend people use our packages which 
are compiled by Go v1.17.3 in production. I think every release of Go 
(43 of them!) since that one has contained a security patch.


Even if you dislike Go, but can work your way through a package, please 
consider signing up!


--
Katherine

Re: Updates for Go

[Development of GNU Guix and the GNU System distribution.]

Hi Katherine,

On Tue, Aug 15, 2023 at 12:59 PM Katherine Cox-Buday
 wrote:
>
> There's also no one on Guix's Go team. I've created a patch to add
> myself[1]

Your courage and initiative are inspiring. (Unfortunately, my skills
are lousy.) Anybody with an interest in Golang should please speak up!

Kind regards
Felix

Updates for Go

[Katherine Cox-Buday]

Hey all,

Our Go ecosystem is currently in need of a lot of love.

* The Go Team

There is currently no branch for Go updates. I know Leo had tried to get 
one setup at one point[0] but ran into issues. I'm unclear if they were 
ever resolved, but the branch isn't there, and we need one.


There's also no one on Guix's Go team. I've created a patch to add 
myself[1], although my availability has historically been very 
intermittent. Please consider joining since the Go ecosystem contains 
some pretty critical packages (e.g. Docker, syncthing, etc.).


* Go Versions

Go v1.21.0 has just been released. Go's release policy[2] states that 
releases are supported until two major releases have been made. We're 
building everything in Go 1.17.3 by default right now which puts our Go 
ecosystem at risk since our packages won't be using security fixes that 
have since been released.


To help address this, I've proposed[3] a Go v1.21.0 package, but not 
made it the default.


Next, we should get started on compiling everything with a supported 
version of Go. It would be nice to switch the default to Go v1.21.0 
since it has some nice performance improvements. To do this, we'll need 
the feature branch so we can begin working on what's broken.


What do you all think? If it's just me trying to fix broken packages, 
we're going to be on Go v1.17 for a long time!


[0] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00097.html
[1] https://issues.guix.gnu.org/65314
[2] https://go.dev/doc/devel/release#policy
[3] https://issues.guix.gnu.org/65317

--
Katherine

Re: core-updates, let’s go!

[Kei Kebreau]

Leo Famulari  writes:

> On Mon, Jan 01, 2018 at 06:12:49PM +0100, Ludovic Courtès wrote:
>> We’ll have to stay focused in the coming days to fix everything as
>> quickly as we can.
>
> The latest core-updates evaluation is coming together [0], but there are
> still a few notable issues:
>
> * Pandas is failing to build:
>
> https://hydra.gnu.org/job/gnu/core-updates/python2-pandas-0.19.2.x86_64-linux
>
> * CLISP is failing to build, which breaks SBCL:
>
> https://hydra.gnu.org/job/gnu/core-updates/clisp-2.49-60.x86_64-linux
>
> * Several packages are failing due to changes in ICU related to type
> definitions. For example, evolution-data-server is failing to build,
> which breaks GNOME:
>
> https://hydra.gnu.org/job/gnu/core-updates/evolution-data-server-3.24.3.x86_64-linux
>
> * ant-bootstrap failed to build, breaking most (all?) of the Java
> packages. This is being discussed in . I
> can't retry the ant-bootstrap build because the package is not public,
> so there is no corresponding job on Hydra. An example:
>
> https://hydra.gnu.org/build/2443807
>
> * And more!
>
> Why not check if the packages you rely on are working on core-updates?
> These are the packages that fail compared to the master branch:
>
> [0] 
> https://hydra.gnu.org/eval/109888?compare=master=1#tabs-now-fail

I have a patch for CLISP that addresses the source of the build
failure. I'm currently testing to see if the resulting binary can build
SBCL properly. Expect to see the patch on the guix-patches list soon!


signature.asc
Description: PGP signature

Re: core-updates, let’s go!

[Leo Famulari]

On Mon, Jan 01, 2018 at 06:12:49PM +0100, Ludovic Courtès wrote:
> We’ll have to stay focused in the coming days to fix everything as
> quickly as we can.

The latest core-updates evaluation is coming together [0], but there are
still a few notable issues:

* Pandas is failing to build:

https://hydra.gnu.org/job/gnu/core-updates/python2-pandas-0.19.2.x86_64-linux

* CLISP is failing to build, which breaks SBCL:

https://hydra.gnu.org/job/gnu/core-updates/clisp-2.49-60.x86_64-linux

* Several packages are failing due to changes in ICU related to type
definitions. For example, evolution-data-server is failing to build,
which breaks GNOME:

https://hydra.gnu.org/job/gnu/core-updates/evolution-data-server-3.24.3.x86_64-linux

* ant-bootstrap failed to build, breaking most (all?) of the Java
packages. This is being discussed in . I
can't retry the ant-bootstrap build because the package is not public,
so there is no corresponding job on Hydra. An example:

https://hydra.gnu.org/build/2443807

* And more!

Why not check if the packages you rely on are working on core-updates?
These are the packages that fail compared to the master branch:

[0] 
https://hydra.gnu.org/eval/109888?compare=master=1#tabs-now-fail


signature.asc
Description: PGP signature

Re: core-updates, let’s go!

[Leo Famulari]

On Wed, Jan 03, 2018 at 01:35:31PM -0500, Leo Famulari wrote:
> Libtiff failed the first build attempt when the upstream FTP server
> timed out. It has succeeded since then, but this will have caused a lot
> of dependent packages to fail.
> 
> Should I "Restart all dependency failed builds" now?

I went ahead and did this because it was too hard to failed packages
that did not fail due to the earlier libtiff problem.


signature.asc
Description: PGP signature

Re: change to the wrap-program procedure (was: core-updates, let’s go!)

[Ricardo Wurmus]

Hi Hartmut,

> Am 31.12.2017 um 18:27 schrieb Ricardo Wurmus:
>> There’s also a change to the wrap-program procedure that does without
>> the shell wrappers and instead prepends a short Guile program, which is
>> read as a comment in the target language.  It’s better to play with this
>> in the next core-updates cycle.
>
> I assume, this is what we discussed in
> .
>
> I just wonder whether this will keep the encoding-lines in place, like
> these:
>
> #!/usr/bin/python
> # -*- coding: latin-1 -*-
> from __future__ import print_function

In the current implementation that I have here it would not.

According to the documentation[1], the regular expression
“coding[=:]\s*([-\w.]+)” must match for a comment on the second line of
a script, so I’d think that the following would still be valid:

--8<---cut here---start->8---
#!/gnu/store/…/bin/guile --no-auto-compile
#!#; -*- coding: latin-1 -*-
#\-(setenv …)
#\-(apply execl …)
#!/gnu/store/…/bin/python3
# -*- coding: latin-1 -*-
…
--8<---cut here---end--->8---

All we need to do is read the second line of the script and see if it is
a coding comment; in that case we’d place the content on the second
line.  I’ll add a special case for this and a test case.  Thanks for
bringing it up!

[1]: 
https://docs.python.org/3/reference/lexical_analysis.html#encoding-declarations

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

change to the wrap-program procedure (was: core-updates, let’s go!)

[Hartmut Goebel]

Am 31.12.2017 um 18:27 schrieb Ricardo Wurmus:
> There’s also a change to the wrap-program procedure that does without
> the shell wrappers and instead prepends a short Guile program, which is
> read as a comment in the target language.  It’s better to play with this
> in the next core-updates cycle.

I assume, this is what we discussed in
.

I just wonder whether this will keep the encoding-lines in place, like
these:

#!/usr/bin/python
# -*- coding: latin-1 -*-
from __future__ import print_function

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: core-updates, let’s go!

[Efraim Flashner]

On Mon, Jan 01, 2018 at 06:12:49PM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> I’ve started a new evaluation, let’s see what happens!
> 
> We’ll have to stay focused in the coming days to fix everything as
> quickly as we can.
> 
> Thanks, and happy new year where applicable!  :-)
> 
> Ludo’.

I've tracked down the gcc@4.[89] build failures, I'll try to fix gcc@4.9
without affecting libstdc++-boot0

-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: core-updates, let’s go!

[Ludovic Courtès]

Hello Guix!

I’ve started a new evaluation, let’s see what happens!

We’ll have to stay focused in the coming days to fix everything as
quickly as we can.

Thanks, and happy new year where applicable!  :-)

Ludo’.

Re: core-updates, let’s go!

[Ricardo Wurmus]

Marius Bakke  writes:

> Danny Milosavljevic  writes:
>
>> I'd like to get the patch
>>
>> [bug#29856] [PATCH core-updates] guix: python-build-system:  Modify 
>> ".py" files in-place.
>>
>> into this core-updates cycle.  What do you all think?
>
> We already have a substantial amount of Python changes on core-updates,
> so we could save ourselves some trouble by adding this when there's less
> "congestion" in case of any problems.

I agree with Marius.

There’s also a change to the wrap-program procedure that does without
the shell wrappers and instead prepends a short Guile program, which is
read as a comment in the target language.  It’s better to play with this
in the next core-updates cycle.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: core-updates, let’s go!

[Marius Bakke]

Danny Milosavljevic  writes:

> I'd like to get the patch
>
> [bug#29856] [PATCH core-updates] guix: python-build-system:  Modify ".py" 
> files in-place.
>
> into this core-updates cycle.  What do you all think?

We already have a substantial amount of Python changes on core-updates,
so we could save ourselves some trouble by adding this when there's less
"congestion" in case of any problems.


signature.asc
Description: PGP signature

Re: core-updates, let’s go!

[Marius Bakke]

Ludovic Courtès  writes:

> Hello Guix!
>
> ‘core-updates’ is doing OK!
>
>   https://hydra.gnu.org/jobset/gnu/core-updates
>
> The remaining issues are GCC 4.8 and 4.9 failing to build, which are
> nothing serious and left as an exercise to the reader.
>
> What about starting a new evaluation of all the packages now so we can
> merge it early 2018?
>
> Efraim, can you confirm that everything’s alright on aarch64?
>
> We’re lacking a couple of glibc commits from the ‘release/2.26/master’
> branch (see below), but Florian Weimer marked them as non-critical
> provided we have the previous fixes.  Should we include them anyway and
> rebuild the whole thing?  (I’m inclined to answer “no”.)

I agree, let's not hold up this branch any longer.  Godspeed!


signature.asc
Description: PGP signature

Re: core-updates, let’s go!

[Danny Milosavljevic]

On Sun, 31 Dec 2017 12:14:16 +0100
Danny Milosavljevic  wrote:

> I'd like to get the patch
> 
> [bug#29856] [PATCH core-updates] guix: python-build-system:  Modify ".py" 
> files in-place.
> 
> into this core-updates cycle.  What do you all think?
> 
> If we don't do it then a lot of Python programs (the ones who use 
> "console_scripts" in setup.py as they should) will not work (because we emit 
> a shell script wrapper and Python tries to import it - that's going to end 
> badly).  On the other hand, I don't know the effect on other programs (read: 
> I didn't build the world with it - but I did build quite a few Python 
> programs).  You can try it with the guix package ledger-agent and try to 
> invoke "ledger-agent".
> 
> The patch only affects files whose names end in ".py" since Python's importer 
> will only find those in any case.  So its possible ill effect is limited.

^^^ Correction: whose names end in ".py" and that end up in "bin" or "sbin".  
Otherwise it would be common.

> 
> Hartmut, what do you think?



-- 
W: https://www.friendly-machines.at/

Re: core-updates, let’s go!

[Danny Milosavljevic]

I'd like to get the patch

[bug#29856] [PATCH core-updates] guix: python-build-system:  Modify ".py" 
files in-place.

into this core-updates cycle.  What do you all think?

If we don't do it then a lot of Python programs (the ones who use 
"console_scripts" in setup.py as they should) will not work (because we emit a 
shell script wrapper and Python tries to import it - that's going to end 
badly).  On the other hand, I don't know the effect on other programs (read: I 
didn't build the world with it - but I did build quite a few Python programs).  
You can try it with the guix package ledger-agent and try to invoke 
"ledger-agent".

The patch only affects files whose names end in ".py" since Python's importer 
will only find those in any case.  So its possible ill effect is limited.

Hartmut, what do you think?

Re: core-updates, let’s go!

[Efraim Flashner]

On Sun, Dec 31, 2017 at 11:27:37AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> ‘core-updates’ is doing OK!
> 
>   https://hydra.gnu.org/jobset/gnu/core-updates
> 
> The remaining issues are GCC 4.8 and 4.9 failing to build, which are
> nothing serious and left as an exercise to the reader.
> 
> What about starting a new evaluation of all the packages now so we can
> merge it early 2018?
> 

I've merged master into core-updates and I'm running 'make' against it
as we speak to make sure I didn't make any obvious errors, and then I'm
about to push it.

> Efraim, can you confirm that everything’s alright on aarch64?
> 

I've been keeping mesa in good shape on aarch64, I'm pretty sure gtk+,
gtk+@2 and qtbase build alright.

> We’re lacking a couple of glibc commits from the ‘release/2.26/master’
> branch (see below), but Florian Weimer marked them as non-critical
> provided we have the previous fixes.  Should we include them anyway and
> rebuild the whole thing?  (I’m inclined to answer “no”.)
> 
> Thanks,
> Ludo’.
> 
> 
> commit 633e2f7f3d88df6427aa3a7a984d3a6b796d9611 (HEAD -> release/2.26/master, 
> origin/release/2.26/master)
> Author: Florian Weimer 
> Date:   Thu Dec 14 15:05:57 2017 +0100
> 
> elf: Count components of the expanded path in _dl_init_path [BZ #22607]
> 
> (cherry picked from commit 3ff3dfa5af313a6ea33f3393916f30eece4f0171)
> 
> commit 43b3cb59b2288953efc26e70fe7c6eb437513b1f
> Author: Florian Weimer 
> Date:   Thu Dec 14 15:18:38 2017 +0100
> 
> elf: Compute correct array size in _dl_init_paths [BZ #22606]
> 
> (cherry picked from commit 8a0b17e48b83e933960dfeb8fa08b259f03f310e)

-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

core-updates, let’s go!

[Ludovic Courtès]

Hello Guix!

‘core-updates’ is doing OK!

  https://hydra.gnu.org/jobset/gnu/core-updates

The remaining issues are GCC 4.8 and 4.9 failing to build, which are
nothing serious and left as an exercise to the reader.

What about starting a new evaluation of all the packages now so we can
merge it early 2018?

Efraim, can you confirm that everything’s alright on aarch64?

We’re lacking a couple of glibc commits from the ‘release/2.26/master’
branch (see below), but Florian Weimer marked them as non-critical
provided we have the previous fixes.  Should we include them anyway and
rebuild the whole thing?  (I’m inclined to answer “no”.)

Thanks,
Ludo’.


commit 633e2f7f3d88df6427aa3a7a984d3a6b796d9611 (HEAD -> release/2.26/master, 
origin/release/2.26/master)
Author: Florian Weimer 
Date:   Thu Dec 14 15:05:57 2017 +0100

elf: Count components of the expanded path in _dl_init_path [BZ #22607]

(cherry picked from commit 3ff3dfa5af313a6ea33f3393916f30eece4f0171)

commit 43b3cb59b2288953efc26e70fe7c6eb437513b1f
Author: Florian Weimer 
Date:   Thu Dec 14 15:18:38 2017 +0100

elf: Compute correct array size in _dl_init_paths [BZ #22606]

(cherry picked from commit 8a0b17e48b83e933960dfeb8fa08b259f03f310e)