Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Federico Beffa]

On Mon, Oct 5, 2015 at 11:09 PM, Ludovic Courtès  wrote:
> Federico Beffa  skribis:
>
>> I'm wondering if it would be better to point GUIX_LOCPATH to
>>
>> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>>
>> and have 'glibc' itself to append it's own version number to that
>> string. In this way pre 2.22 programs (last official Guix(SD) release)
>> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'.
>
> With the discussion on IRC following your message, I ended up going back
> to the solution that appends /2.22 to the GUIX_LOCPATH entries
> automatically:
>
>   http://article.gmane.org/gmane.comp.gnu.guix.devel/12004
>
> We’ll settle on that and declare the problem solved now.  :-)
>
> Sorry for the hesitations.  It’s one of these situations where we have
> conflicting principles: simplifying the users lives, and remaining close
> to what upstream does.

I think this is the right thing to do. It's not just about making the
lives of users easier, but rather preventing that users will see their
profile break after an update (when a new 'glibc' with incompatible
locales will come out). IMO such a behavior would just be bad design.

Thanks,
Fede

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Federico Beffa  skribis:

> On Mon, Oct 5, 2015 at 4:35 PM, Ludovic Courtès  wrote:
>> Mark H Weaver  skribis:
>>
>>> l...@gnu.org (Ludovic Courtès) writes:
>>>
 So with current ‘core-updates’, someone on a “foreign distro” needs to
 do:

   guix package -i glibc-locales
   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22

 Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
 of an annoyance for end users, but the point is that eventually this
 would allow us to recommend things like:

   export 
 GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

 The only question is whether having the “/2.22” prefix by default is a
 good idea.  Opinions?
>>>
>>> I think the "/2.22" suffix will be needed to prevent another awkward
>>> transition the next time glibc makes an incompatible change to their
>>> locales.  Suppose that 2.23 makes another incompatible change.  After
>>> that, many Guix systems will have a mixture of software linked with
>>> glibc-2.22 and glibc-2.23.
>>
>> Yes.  But we could just as well have ‘glibc-utf8-locales’ where
>> everything is in lib/locale directly, and ‘glibc-transition-locales’
>> where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.
>
> I'm wondering if it would be better to point GUIX_LOCPATH to
>
> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>
> and have 'glibc' itself to append it's own version number to that
> string. In this way pre 2.22 programs (last official Guix(SD) release)
> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'. The
> next releases (with 'glibc-2.22' and following) will provide locales
> in versioned sub-directory (no conflict with pre 2.22) and the new
> programs will co-exist without users ever having to modify
> GUIX_LOCPATH.

This is what the patch at
 does.

However, it has the drawback of being fairly intrusive and non-trivial.
Intrusive in that it touches libc, and if it doesn’t make it upstream,
then we’ll have difficulties dealing with it.  Non-trivial in that I
think it’s pretty unusual for software to automatically modify search
path entries (when defining FOOPATH=/x:/y, one expects Foo to look under
/x and /y, not under /x/2.22.)

None of these two approaches is perfect, but I’m inclined to make the
choice that requires the smallest amount of modifications to libc.

Thoughts?

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Federico Beffa  skribis:

> I'm wondering if it would be better to point GUIX_LOCPATH to
>
> export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
>
> and have 'glibc' itself to append it's own version number to that
> string. In this way pre 2.22 programs (last official Guix(SD) release)
> would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'.

With the discussion on IRC following your message, I ended up going back
to the solution that appends /2.22 to the GUIX_LOCPATH entries
automatically:

  http://article.gmane.org/gmane.comp.gnu.guix.devel/12004

We’ll settle on that and declare the problem solved now.  :-)

Sorry for the hesitations.  It’s one of these situations where we have
conflicting principles: simplifying the users lives, and remaining close
to what upstream does.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver  skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> So with current ‘core-updates’, someone on a “foreign distro” needs to
>> do:
>>
>>   guix package -i glibc-locales
>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>>
>> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
>> of an annoyance for end users, but the point is that eventually this
>> would allow us to recommend things like:
>>
>>   export 
>> GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>
>> The only question is whether having the “/2.22” prefix by default is a
>> good idea.  Opinions?
>
> I think the "/2.22" suffix will be needed to prevent another awkward
> transition the next time glibc makes an incompatible change to their
> locales.  Suppose that 2.23 makes another incompatible change.  After
> that, many Guix systems will have a mixture of software linked with
> glibc-2.22 and glibc-2.23.

Yes.  But we could just as well have ‘glibc-utf8-locales’ where
everything is in lib/locale directly, and ‘glibc-transition-locales’
where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.

> One question: when this happens, do we have a mechanism for
> automatically setting GUIX_LOCPATH to:
>
>   $HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> or will that have to be done manually?

Unless glibc is in the profile, that has to be done manually.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver  skribis:

> l...@gnu.org (Ludovic Courtès) writes:
>
>> After a lot more thought, I changed my mind.
>>
>> I realized that with the patch at
>> , it’s OK to
>> have, say,
>>
>>   
>> LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>
>> That way, programs will pick locale data that is compatible; so a 2.23
>> program might pick most of its locale data from the /2.22 directory (if
>> they are compatible), and some of them from /2.23 (if the format has
>> changed.)
>
> I'm a bit uncomfortable with this.  A few questions:
>
> * Will your glibc patch reliably ensure that no functionality is lost
>   because of picking up the wrong version of locales?

If locale data is missing or incompatible, ‘setlocale’ returns ENOENT or
EINVAL.  That’s the worst that can happen.

> * Do we have a mechanism for automatically setting LOCPATH (or
>   GUIX_LOCPATH) to include all of the available locale versioned
>   subdirectories?

No.

This is not a problem on GuixSD because we would populate
/run/current-system/locale/{2.22,2.23} by default and rarely need to
fiddle with GUIX_LOCPATH.

On foreign distros, it means people have to maintain GUIX_LOCPATH by
hand; this is already the case, but now they’d have to append “/2.22”
and possibly other subdirectories.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Federico Beffa]

On Mon, Oct 5, 2015 at 4:35 PM, Ludovic Courtès  wrote:
> Mark H Weaver  skribis:
>
>> l...@gnu.org (Ludovic Courtès) writes:
>>
>>> So with current ‘core-updates’, someone on a “foreign distro” needs to
>>> do:
>>>
>>>   guix package -i glibc-locales
>>>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>>>
>>> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
>>> of an annoyance for end users, but the point is that eventually this
>>> would allow us to recommend things like:
>>>
>>>   export 
>>> GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>>>
>>> The only question is whether having the “/2.22” prefix by default is a
>>> good idea.  Opinions?
>>
>> I think the "/2.22" suffix will be needed to prevent another awkward
>> transition the next time glibc makes an incompatible change to their
>> locales.  Suppose that 2.23 makes another incompatible change.  After
>> that, many Guix systems will have a mixture of software linked with
>> glibc-2.22 and glibc-2.23.
>
> Yes.  But we could just as well have ‘glibc-utf8-locales’ where
> everything is in lib/locale directly, and ‘glibc-transition-locales’
> where things are under lib/locale/2.22 and lib/locale/2.23.  Dunno.

I'm wondering if it would be better to point GUIX_LOCPATH to

export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale

and have 'glibc' itself to append it's own version number to that
string. In this way pre 2.22 programs (last official Guix(SD) release)
would find pre-2.22 locale in '/$HOME/.guix-profile/lib/locale'. The
next releases (with 'glibc-2.22' and following) will provide locales
in versioned sub-directory (no conflict with pre 2.22) and the new
programs will co-exist without users ever having to modify
GUIX_LOCPATH.

Regards,
Fede

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Daniel Pimentel  skribis:

> I added 'export LC_ALL="C"' in my ~.bashrc.
>
> I'll to do upgrade and report it.

With LC_ALL=C, one sidesteps the problem.  For real testing, you need to
use LC_ALL=pt_BR.utf8 or something like that.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

l...@gnu.org (Ludovic Courtès) writes:

> So with current ‘core-updates’, someone on a “foreign distro” needs to
> do:
>
>   guix package -i glibc-locales
>   export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22
>
> Note the extra “/2.22”, which comes from commit f2d7bbb.  This is a bit
> of an annoyance for end users, but the point is that eventually this
> would allow us to recommend things like:
>
>   export 
> GUIX_LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> The only question is whether having the “/2.22” prefix by default is a
> good idea.  Opinions?

I think the "/2.22" suffix will be needed to prevent another awkward
transition the next time glibc makes an incompatible change to their
locales.  Suppose that 2.23 makes another incompatible change.  After
that, many Guix systems will have a mixture of software linked with
glibc-2.22 and glibc-2.23.

One question: when this happens, do we have a mechanism for
automatically setting GUIX_LOCPATH to:

  $HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

or will that have to be done manually?

 Mark

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

l...@gnu.org (Ludovic Courtès) writes:

> After a lot more thought, I changed my mind.
>
> I realized that with the patch at
> , it’s OK to
> have, say,
>
>   
> LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23
>
> That way, programs will pick locale data that is compatible; so a 2.23
> program might pick most of its locale data from the /2.22 directory (if
> they are compatible), and some of them from /2.23 (if the format has
> changed.)

I'm a bit uncomfortable with this.  A few questions:

* Will your glibc patch reliably ensure that no functionality is lost
  because of picking up the wrong version of locales?

* Do we have a mechanism for automatically setting LOCPATH (or
  GUIX_LOCPATH) to include all of the available locale versioned
  subdirectories?

  Mark

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

After a lot more thought, I changed my mind.

I realized that with the patch at
, it’s OK to
have, say,

  
LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

That way, programs will pick locale data that is compatible; so a 2.23
program might pick most of its locale data from the /2.22 directory (if
they are compatible), and some of them from /2.23 (if the format has
changed.)

We still need ‘GUIX_LOCPATH’ to avoid the assertion failure with older
libcs, only without the complicated semantics discussed yesterday.

The result is in ‘core-updates’:

  b6ac545 * gnu: commencement: Remove the tricky locale compatibility handling.
  f2d7bbb * gnu: glibc: Look for locale data in versioned sub-directories.
  fbb909a * gnu: glibc: Honor 'GUIX_LOCPATH'.

Ludo’.

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Daniel Pimentel]

On 2015-10-03 19:20, l...@gnu.org wrote:

After a lot more thought, I changed my mind.

I realized that with the patch at
, it’s OK 
to

have, say,


LOCPATH=$HOME/.guix-profile/lib/locale/2.22:$HOME/.guix-profile/lib/locale/2.23

That way, programs will pick locale data that is compatible; so a 2.23
program might pick most of its locale data from the /2.22 directory (if
they are compatible), and some of them from /2.23 (if the format has
changed.)

We still need ‘GUIX_LOCPATH’ to avoid the assertion failure with older
libcs, only without the complicated semantics discussed yesterday.

The result is in ‘core-updates’:

  b6ac545 * gnu: commencement: Remove the tricky locale compatibility 
handling.
  f2d7bbb * gnu: glibc: Look for locale data in versioned 
sub-directories.

  fbb909a * gnu: glibc: Honor 'GUIX_LOCPATH'.

Ludo’.

Thanks!

I added 'export LC_ALL="C"' in my ~.bashrc.

I'll to do upgrade and report it.
--
Daniel Pimentel (d4n1 3:)

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

Mark H Weaver  skribis:

> For compatibility reasons, I think we should add "/2.22" _only_ to the
> entries of GUIX_LOCPATH.  IMO, it's unwise to change the meaning of
> LOCPATH.  Some programs, build systems, or user scripts might use
> "localedef" in a directory, set LOCPATH to that directory, and expect it
> to work.

Yes, makes sense.  I noticed that the recipe for ‘tre’ relies on
‘LOCPATH’, and I agree that it’s best to keep ‘LOCPATH’ semantics.

So, as discussed on IRC, here’s a second patch that:

  • introduces ‘GUIX_LOCPATH’, which is like ‘LOCPATH’, except
versioned;

  • keeps ‘LOCPATH’, without appending “/X.Y” to its entries, and giving
it less precedence than ‘GUIX_LOCPATH’;

We would tell people to migrate away from ‘LOCPATH’ in favor of
‘GUIX_LOCPATH’.

I think we should use this patch as a starting point for a discussion
for the libc people.  I’d argue that would be acceptable even for
upstream (with s/GUIX_LOCPATH/LOCALE_PATH/ for instance.)

Please review carefully.  Ideally I’d like to push this patch or a
variant thereof tomorrow, so we can start the full rebuild.

TIA!

Ludo’.

>From 7258d8c6e2a464f911723cabc2e35f7e05c49192 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= 
Date: Thu, 1 Oct 2015 21:32:50 +0200
Subject: [PATCH] gnu: glibc: Look for locale data in versioned
 sub-directories.

* gnu/packages/patches/glibc-versioned-locpath.patch: New file.
* gnu-system.am (dist_patch_DATA): Add it.
* gnu/packages/base.scm (glibc)[source]: Use it.
  [arguments]: Add explicit version sub-directory to
  libc_cv_localedir.
  [native-search-paths]: Use 'GUIX_LOCPATH' instead of 'LOCPATH'.
  (glibc-locales, glibc-utf8-locales): Write to a VERSION
  sub-directory.
---
 doc/guix.texi  |  23 +-
 gnu-system.am  |   1 +
 gnu/packages/base.scm  |  17 +-
 gnu/packages/patches/glibc-versioned-locpath.patch | 240 +
 4 files changed, 268 insertions(+), 13 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 68ee451..1ad4229 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -933,24 +933,24 @@ daemons on the same machine.
 @node Application Setup
 @section Application Setup
 
-When using Guix on top of GNU/Linux distribution other than GuixSD, a
-few additional steps are needed to get everything in place.  Here are
-some of them.
+When using Guix on top of GNU/Linux distribution other than GuixSD---a
+so-called @dfn{foreign distro}---a few additional steps are needed to
+get everything in place.  Here are some of them.
 
 @subsection Locales
 
 @anchor{locales-and-locpath}
 @cindex locales, when not on GuixSD
 @vindex LOCPATH
+@vindex GUIX_LOCPATH
 Packages installed @i{via} Guix will not use the host system's locale
 data.  Instead, you must first install one of the locale packages
-available with Guix and then define the @code{LOCPATH} environment
-variable (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C Library
-Reference Manual}):
+available with Guix and then define the @code{GUIX_LOCPATH} environment
+variable:
 
 @example
 $ guix package -i glibc-locales
-$ export LOCPATH=$HOME/.guix-profile/lib/locale
+$ export GUIX_LOCPATH=$HOME/.guix-profile/lib/locale
 @end example
 
 Note that the @code{glibc-locales} package contains data for all the
@@ -958,6 +958,15 @@ locales supported by the GNU@tie{}libc and weighs in at around
 110@tie{}MiB.  Alternately, the @code{glibc-utf8-locales} is smaller but
 limited to a few UTF-8 locales.
 
+The @code{GUIX_LOCPATH} variable plays the exact same role as
+@code{LOCPATH} (@pxref{Locale Names, @code{LOCPATH},, libc, The GNU C
+Library Reference Manual}).  However, since it is honored only by Guix's
+libc, and not by the libc provided by foreign distros, using
+@code{GUIX_LOCPATH} allows you to make sure the the foreign distro's
+programs will not end up loading incompatible locale data.  This is
+important because the locale data format used by different libc versions
+may be incompatible.
+
 @subsection X11 Fonts
 
 The majority of graphical applications use Fontconfig to locate and
diff --git a/gnu-system.am b/gnu-system.am
index f2f7e17..571e660 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -472,6 +472,7 @@ dist_patch_DATA =		\
   gnu/packages/patches/glibc-ldd-x86_64.patch			\
   gnu/packages/patches/glibc-locales.patch			\
   gnu/packages/patches/glibc-o-largefile.patch			\
+  gnu/packages/patches/glibc-versioned-locpath.patch		\
   gnu/packages/patches/gmp-arm-asm-nothumb.patch		\
   gnu/packages/patches/gnucash-price-quotes-perl.patch		\
   gnu/packages/patches/gnutls-doc-fix.patch			\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index a7d9459..1402a44 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -476,6 +476,7 @@ store.")
 (modules '((guix 

[PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Ludovic Courtès]

So, below are a couple of patches implementing ideas that were discussed here
and on IRC to improve the locale mess.

The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
that on foreign distros, users could set this variable instead of 'LOCPATH',
and the host distro's programs wouldn't break.

The second patch was suggested by Mark.  It basically adds "/2.22" to every
directory name specified in LOCPATH, as well as to the default locale
directory.  The idea here is that libc would only stumble on compatible
locale data.

Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
effectively Guix's libc would already be interpreting 'LOCPATH' differently.
So my inclination would be to apply only the second one.

Thoughts?

I'd like to get that on 'core-updates' soon so we can quickly get the better
fix.

Thanks,
Ludo'.

Ludovic Courtès (2):
  gnu: glibc: Honor 'GUIX_LOCPATH'.
  gnu: glibc: Look for locale data in versioned sub-directories.

 doc/guix.texi  | 23 +++
 gnu-system.am  |  2 +
 gnu/packages/base.scm  | 24 ++-
 gnu/packages/patches/glibc-guix-locpath.patch  | 38 +
 gnu/packages/patches/glibc-versioned-locpath.patch | 47 ++
 5 files changed, 116 insertions(+), 18 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-guix-locpath.patch
 create mode 100644 gnu/packages/patches/glibc-versioned-locpath.patch

-- 
2.5.0

Re: [PATCH 0/2] Avoiding incompatible locale data in LOCPATH

[Mark H Weaver]

Ludovic Courtès  writes:

> So, below are a couple of patches implementing ideas that were discussed here
> and on IRC to improve the locale mess.
>
> The first patch adds the 'GUIX_LOCPATH' environment variable, the idea being
> that on foreign distros, users could set this variable instead of 'LOCPATH',
> and the host distro's programs wouldn't break.
>
> The second patch was suggested by Mark.  It basically adds "/2.22" to every
> directory name specified in LOCPATH, as well as to the default locale
> directory.  The idea here is that libc would only stumble on compatible
> locale data.
>
> Actually with that second patch, I think 'GUIX_LOCPATH' is unneeded, because
> effectively Guix's libc would already be interpreting 'LOCPATH' differently.
> So my inclination would be to apply only the second one.
>
> Thoughts?

For compatibility reasons, I think we should add "/2.22" _only_ to the
entries of GUIX_LOCPATH.  IMO, it's unwise to change the meaning of
LOCPATH.  Some programs, build systems, or user scripts might use
"localedef" in a directory, set LOCPATH to that directory, and expect it
to work.

  Mark