Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi again,

Syncing /var/cache has finally finished, after copying 16 TiB from the
old array in about 16 days:

rsync warning: some files vanished before they could be transferred (code 24) 
at main.c(1330) [sender=3.2.3]

real17019m52.264s
user147m58.061s
sys 686m3.528s

I've now pushed d4b73e1 to guix-artwork which is a blog post about the
sunsetting of gzip substitutes, planned for next month, which should
allow to shave 6.5 TiB of that amount.

Feel free to edit it to correct any typo or other mishaps I may have
made.

With this, we have about 3 weeks to devise the script needed to sanitize
the narinfo files.

Thank you,

Maxim

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi Ricardo and Mathieu,

Ricardo Wurmus  writes:

> Hi Mathieu,
>
> thanks for the patch!
>
>> From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
>> From: Mathieu Othacehe 
>> Date: Tue, 8 Feb 2022 14:28:56 +0100
>> Subject: [PATCH 1/1] store: Warn about daemon deprecation.
>>
>> * guix/deprecation.scm (warn-about-old-daemon): New procedure.
>> * guix/store.scm (build-things): Use it to warn about old daemons.
>> ---
>>  guix/deprecation.scm | 7 +++
>>  guix/store.scm   | 2 ++
>>  2 files changed, 9 insertions(+)
>>
>> diff --git a/guix/deprecation.scm b/guix/deprecation.scm
>> index c66c9367f6..666e178d75 100644
>> --- a/guix/deprecation.scm
>> +++ b/guix/deprecation.scm
>> @@ -1,5 +1,6 @@
>>  ;;; GNU Guix --- Functional package management for GNU
>>  ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès 
>> +;;; Copyright © 2021 Mathieu Othacehe 
>>  ;;;
>>  ;;; This file is part of GNU Guix.
>>  ;;;
>> @@ -24,6 +25,8 @@ (define-module (guix deprecation)
>>  
>>  define-deprecated/public
>>  define-deprecated/alias
>> +
>> +warn-about-old-daemon
>>  warn-about-deprecation))
>>  
>>  ;;; Commentary:
>> @@ -32,6 +35,10 @@ (define-module (guix deprecation)
>>  ;;;
>>  ;;; Code:
>>  
>> +(define (warn-about-old-daemon)
>> +  (warning (G_ "Your Guix daemon is seriously outdated, please consider
>> + updating it by following the 'Upgrading Guix' documentation section.~%")))
>> +
>
> s/documention section/section in the manual/
>
> Should it also mention that staying with the old daemon means that
> binaries will no longer be available?  For example:
>
>Your Guix daemon is no longer supported and will soon no longer be
>able to download binary substitutes.
>
> I realize that what you proposed is a generic warning, but perhaps it’s
> better to mention a direct consequence of failing to upgrade.  (I know
> many people who will not upgrade unless they absolutely have to.)

Good idea; I modified it slightly like so:

--8<---cut here---start->8---
@@ -36,8 +36,9 @@ (define-module (guix deprecation)
 ;;; Code:
 
 (define (warn-about-old-daemon)
-  (warning (G_ "Your Guix daemon is seriously outdated, please consider
- updating it by following the 'Upgrading Guix' documentation section.~%")))
+  (warning (G_ "Your Guix daemon is severely outdated, and will soon cease to
+be able to download binary substitutes.  To upgrade it, refer to the
+'Upgrading Guix' section in the manual.~%")))
 
 (define* (warn-about-deprecation variable properties
  #:key replacement)
--8<---cut here---end--->8---
  
>> --- a/guix/store.scm
>> +++ b/guix/store.scm
>> @@ -1442,6 +1442,8 @@ (define build-things
>>   things)))
>>  (parameterize ((current-store-protocol-version
>>  (store-connection-version store)))
>> +  (when (< (current-store-protocol-version) 355) ;0x163
>> +(warn-about-old-daemon))
>>(if (>= (store-connection-minor-version store) 15)
>>(build store things mode)
>>(if (= mode (build-mode normal))
>
> Maybe add a comment here to explain why this particular protocol version
> is considered old.

I added the following:

--8<---cut here---start->8---
@@ -1442,7 +1442,11 @@ (define build-things
  things)))
 (parameterize ((current-store-protocol-version
 (store-connection-version store)))
-  (when (< (current-store-protocol-version) 355) ;0x163
+  (when (< (current-store-protocol-version) #x163)
+;; This corresponds to the first version bump of the daemon
+;; since the introduction of lzip compression support.  The
+;; version change happened with commit 6ef61cc4c30 on the
+;; 2018/10/15).
 (warn-about-old-daemon))
   (if (>= (store-connection-minor-version store) 15)
   (build store things mode)
--8<---cut here---end--->8---

And pushed with commit 96d7535b03.

Thanks for the feedback!

Maxim

Re: Maven Build System Dependency Issue

[Julien Lepiller]

Hi Phil,

I have already seen the issue previously, and I believe it is now fixed
on master. As you can see here,
https://github.com/guix-mirror/guix/blob/master/gnu/packages/java.scm#L7899
we now propagate the correct parent pom.

In particular, java-jmh is a much more complex package, and it builds
fine with the maven-build-system. So I suggest you to pull to the
latest master commit and enjoy :)

Don't hesitate to get back if you still have issues on master, I'll be
happy to help!

Le Tue, 8 Feb 2022 15:27:14 +, Phil Beadling 
a écrit :

> Hi Guixers,
> 
> First let me say I'm a Maven novice, so it's possible I'm doing
> something dumb on the Maven side of things.
> 
> I'm unable to make a bare-bones Maven project build in Guix.

Re: Dropping gzip-compressed substitutes

[Ricardo Wurmus]

Hi Mathieu,

thanks for the patch!

> From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe 
> Date: Tue, 8 Feb 2022 14:28:56 +0100
> Subject: [PATCH 1/1] store: Warn about daemon deprecation.
>
> * guix/deprecation.scm (warn-about-old-daemon): New procedure.
> * guix/store.scm (build-things): Use it to warn about old daemons.
> ---
>  guix/deprecation.scm | 7 +++
>  guix/store.scm   | 2 ++
>  2 files changed, 9 insertions(+)
>
> diff --git a/guix/deprecation.scm b/guix/deprecation.scm
> index c66c9367f6..666e178d75 100644
> --- a/guix/deprecation.scm
> +++ b/guix/deprecation.scm
> @@ -1,5 +1,6 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès 
> +;;; Copyright © 2021 Mathieu Othacehe 
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -24,6 +25,8 @@ (define-module (guix deprecation)
>  
>  define-deprecated/public
>  define-deprecated/alias
> +
> +warn-about-old-daemon
>  warn-about-deprecation))
>  
>  ;;; Commentary:
> @@ -32,6 +35,10 @@ (define-module (guix deprecation)
>  ;;;
>  ;;; Code:
>  
> +(define (warn-about-old-daemon)
> +  (warning (G_ "Your Guix daemon is seriously outdated, please consider
> + updating it by following the 'Upgrading Guix' documentation section.~%")))
> +

s/documention section/section in the manual/

Should it also mention that staying with the old daemon means that
binaries will no longer be available?  For example:

   Your Guix daemon is no longer supported and will soon no longer be
   able to download binary substitutes.

I realize that what you proposed is a generic warning, but perhaps it’s
better to mention a direct consequence of failing to upgrade.  (I know
many people who will not upgrade unless they absolutely have to.)

> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -1442,6 +1442,8 @@ (define build-things
>   things)))
>  (parameterize ((current-store-protocol-version
>  (store-connection-version store)))
> +  (when (< (current-store-protocol-version) 355) ;0x163
> +(warn-about-old-daemon))
>(if (>= (store-connection-minor-version store) 15)
>(build store things mode)
>(if (= mode (build-mode normal))

Maybe add a comment here to explain why this particular protocol version
is considered old.

-- 
Ricardo

Maven Build System Dependency Issue

[Phil Beadling]

Hi Guixers,

First let me say I'm a Maven novice, so it's possible I'm doing something
dumb on the Maven side of things.

I'm unable to make a bare-bones Maven project build in Guix.

This looks to be a problem with mismatched dependencies in Guix around
java-commons-codec.

I suggest what probably needs to be fixed upstream below (assuming my hunch
is correct!).

However I'm looking for advice on how to workaround this problem today to
get a basic Maven package building.

*My method:*

Following advice here:
https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

Using an environment like so:
guix environment --ad-hoc maven

I create a new git repo and put this in the root (so pom.xml, etc, sit in
in the root of the repo)

mvn archetype:generate -DgroupId=com.quantile.app
-DartifactId=java-test-repo
-DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4
-DinteractiveMode=false



I'm able to package and run everything OK (obviously the dependencies are
being pulled from the web in this case):
mvn package
java -cp target/java-test-repo-1.0-SNAPSHOT.jar com.quantile.app.App

Next I try to create a bare-bones Guix package to build this (this time
there will be no web access during the build):

(define-public java-test-repo-integration
  (let ((commit-integration "b9e3894aa3629fc2d60ceadea2f655d4cb6a826b"))
(package
  (name "java-test-repo")
  (version integration-version)
  (source
   (git-checkout
(url "ssh://git@vcs:1234/ea/java-test-repo.git")
(commit commit-integration)))
  (build-system maven-build-system)
  (home-page "https://vcs/projects/EA/repos/java-test-repo/browse;)
  (synopsis "Java Test Repo Production")
  (description "Java/Maven bare-bones framework for testing")
  (license license-quantile

I get the following build failure:

[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time:  1.136 s
[INFO] Finished at: 2022-02-08T14:39:36Z
[INFO]

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test)
on project java-test-repo: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test failed: Plugin
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4 or one of its
dependencies could not be resolved: Failed to collect dependencies at
org.apache.maven.plugins:maven-surefire-plugin:jar:3.0.0-M4 ->
org.apache.maven.surefire:maven-surefire-common:jar:3.0.0-M4 ->
org.apache.maven.surefire:surefire-api:jar:3.0.0-M4 ->
*commons-codec:commons-codec:jar:1.15:
Failed to read artifact descriptor for
commons-codec:commons-codec:jar:1.15: *Cannot access central (
https://repo.maven.apache.org/maven2) in offline mode and the artifact
org.apache.commons:commons-parent:pom:52 has not been downloaded from it
before. -> [Help 1]


After doing some digging I think I know why:

$ guix show java-commons-codec

name: java-commons-codec

version: 1.15

outputs: out

systems: x86_64-linux i686-linux

dependencies: *apache-commons-parent-pom@50* java-commons-lang3@3.9
java-junit@4.12


So it's packaged with a propagated apache-commons-parent-pom at version 50.


See also here for package definition:

https://github.com/guix-mirror/guix/blob/9fe48723c8266ffe8e6a639be2ec2e362cf20cb5/gnu/packages/java.scm#L7671here


But if I look at the POM in the Gnu Store - it is showing version 52 for
the commons-parent which I am assuming (perhaps incorrectly?) should match
the version 50 above?


/gnu/store/7qgi821xidn9nbb07qw35g7wmgv84jln-java-commons-codec-1.15/lib/m2/./commons-codec/commons-codec/1.15/commons-codec-1.15.pom



  

org.apache.commons

commons-parent

*52*

  


Likewise if I look at the source code on Apache's website - it's 52 too:

https://dlcdn.apache.org//commons/codec/source/commons-codec-1.15-src.tar.gz




  

org.apache.commons

commons-parent

*52*

  


What I think has happened is that java-commons-codec has been promoted from
version 1.14 to version 1.15 without changing the propoagated input?

On version 1.14 everything would match:

https://archive.apache.org/dist/commons/codec/source/commons-codec-1.14-src.tar.gz

  

org.apache.commons

commons-parent

50

  

*So assuming this isn't going to cause an avalanche of other issues the
solution proper seems to be either downgrade java-common-codec to version
1.14, or upgrade the dependency on apache-commons-parent-pom to 52?*

What I'm looking for today is any workaround which allows me to use Maven
without having to wait to for upstream fix.

My first attempt was to try - adding an inherited package and then swapping
this in place of 1.15:

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Maxim,

Sound like a fine plan.

> 1.  Promptly set up both a blog post and a NEWS entry announcing the
> support for gzip substitutes is about to be phased out from the build
> farm (1 month notice), urging users to upgrade their daemon to a version
>>= 1.1.0.

I addition, we could warn users with old daemons as proposed by Ludo &
Ricardo. There's an attached patch here.

> -   ;; TODO: Eventually, disable gzip, as discussed at
> -   ;; 
> .
> -   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
> +   (compression '(("lzip" 9) ("zstd" 19)))
> (cache-bypass-threshold cache-bypass-threshold)
> (workers publish-workers)))

Nice.

> 3. Come up with a Guile script that is able to
>
>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>   files
>   b) recompute and update their 'Signature' field.
>
> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.

I hope the script will be able to complete within a reasonable amount of
time, but we cannot know before trying it out :).

Thanks,

Mathieu
>From bd306a8b20f2033d67755dd332a5d33b2f6b822d Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe 
Date: Tue, 8 Feb 2022 14:28:56 +0100
Subject: [PATCH 1/1] store: Warn about daemon deprecation.

* guix/deprecation.scm (warn-about-old-daemon): New procedure.
* guix/store.scm (build-things): Use it to warn about old daemons.
---
 guix/deprecation.scm | 7 +++
 guix/store.scm   | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/guix/deprecation.scm b/guix/deprecation.scm
index c66c9367f6..666e178d75 100644
--- a/guix/deprecation.scm
+++ b/guix/deprecation.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2019, 2020, 2021 Ludovic Courtès 
+;;; Copyright © 2021 Mathieu Othacehe 
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +25,8 @@ (define-module (guix deprecation)
 
 define-deprecated/public
 define-deprecated/alias
+
+warn-about-old-daemon
 warn-about-deprecation))
 
 ;;; Commentary:
@@ -32,6 +35,10 @@ (define-module (guix deprecation)
 ;;;
 ;;; Code:
 
+(define (warn-about-old-daemon)
+  (warning (G_ "Your Guix daemon is seriously outdated, please consider
+ updating it by following the 'Upgrading Guix' documentation section.~%")))
+
 (define* (warn-about-deprecation variable properties
  #:key replacement)
   (let ((location (and properties (source-properties->location properties
diff --git a/guix/store.scm b/guix/store.scm
index a93e9596d9..11dfc095b1 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -1442,6 +1442,8 @@ (define build-things
  things)))
 (parameterize ((current-store-protocol-version
 (store-connection-version store)))
+  (when (< (current-store-protocol-version) 355) ;0x163
+(warn-about-old-daemon))
   (if (>= (store-connection-minor-version store) 15)
   (build store things mode)
   (if (= mode (build-mode normal))
-- 
2.34.0

Re: Dropping gzip-compressed substitutes

[Maxim Cournoyer]

Hi Mathieu!

Mathieu Othacehe  writes:

> Hey Maxim,
>
> Sound like a fine plan.
>
>> 1.  Promptly set up both a blog post and a NEWS entry announcing the
>> support for gzip substitutes is about to be phased out from the build
>> farm (1 month notice), urging users to upgrade their daemon to a version
>>>= 1.1.0.
>
> I addition, we could warn users with old daemons as proposed by Ludo &
> Ricardo. There's an attached patch here.

Awesome, thank you!  It LGTM.  Could you apply it?

>> -   ;; TODO: Eventually, disable gzip, as discussed at
>> -   ;; 
>> .
>> -   (compression '(("gzip" 9) ("lzip" 9) ("zstd" 19)))
>> +   (compression '(("lzip" 9) ("zstd" 19)))
>> (cache-bypass-threshold cache-bypass-threshold)
>> (workers publish-workers)))
>
> Nice.
>
>> 3. Come up with a Guile script that is able to
>>
>>   a) Strip gzip-related metadata from the .narinfo guix-publish metadata
>>   files
>>   b) recompute and update their 'Signature' field.
>>
>> 4. Finally, 'rm -r /var/guix/publish/gzip' and free about 6.5 TiB of data.
>
> I hope the script will be able to complete within a reasonable amount of
> time, but we cannot know before trying it out :).

The script will run on the cache data kept on the SSD array (which we
can snapshot before experimenting with). It'll probably take a while
still, so care should be taken to ensure it takes as little time as
possible (e.g. spawning many threads or processes to spread the load
across multiple CPUs).

I'll work on the blog post and news entry, and then we'll have to look
at the script.

Thank you!

Maxim

Rename glibc-utf8-locales to glibc-limited-utf8-locales

[Rovanion Luckey]

Hi,
I have on different occasions had the issue of the package manager Guix
hinting that locales were not installed or that GUIX_LOCPATH was not set,
even though they were. Today I went to the bottom of the issue [0] and have
found what in my opinion is the root cause:

Despite its name, the package glibc-utf8-locales does not contain all
utf8-locales. There is nothing in the name that makes this apparent to the
user, especially given that the package named glibc-locales contains all
locales.

My suggestion then is to change this package name from glibc-utf8-locales
to glibc-limited-utf8-locales to make this distinction apparent to the
user.

The package name was mentioned enough times in the code of guix that I
hesitate to submit a patch with
s/glibc-utf8-locales/glibc-limited-utf8-locales/ run over the codebase. But
perhaps it is that easy.

[0]: https://unix.stackexchange.com/questions/689761