Re: [Perl/locales] Warning about locales
--- Philip Guenther [EMAIL PROTECTED] wrote: pOn Sun, Jun 15, 2008 at 12:46 PM, F. Caulier [EMAIL PROTECTED] wrote: --- Pieter Verberne [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 07:20:32AM -0700, F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). So something has set LC_CTYPE to en_US.UTF-8, a locale which is not supported by OpenBSD, which results in the setlocale() call failing, something perl complains about so that you don't blame perl when you get broken results. ... I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: UTF-8 uses variable-length characters while ISO8859-* uses fixed-width (8bit) characters. Giving the locale calls the same data for those two is likely to result in incorrect behavior for all characters 127. Wouldn't it be better to simply not lie and just set the locale to en_US.ISO8859-1? The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. Well, have you examined all the programs involved in getting to that shell inside xterm to see if any of them document that they alter the environment? For example, if you use 'uxterm' instead of 'xterm' then you'll see exactly the above behavior, as uxterm is just a script that sets LC_CTYPE=en_US.UTF-8 and then invokes xterm with the -en option. If nothing obvious sticks out, consider debugging further by checking the environment seen by your .xsession (if you xdm) by adding a line like this: env $HOME/.xsession-env.out to it. Similarly, check the shell's environment by doing something similar from your .profile. I tried to figure out why this problem occurs and following to that I noticed that this perllocale warning only comes up when dropping a pkg_* directly in xterm. When using screen in an xterm and dropping pkg_* to it everything will work fine. Same for tty shells without X where everything works fine too. Windows inside screen inherit their environment from the original screen process. So, how do you start the initial (daemon) screen process? From outside X, before running xinitrc? From your .xinitrc or .xsession? From an xterm? I don't know much about this terminal stuff, but if everything beside XTerm works fine, could it be that XTerm itself and not the locales are the problems' source? Maybe XTerm doesn't manage to pass on the locales correctly? Something is setting LC_CTYPE to an unsupported value. That's the program that needs to be fixed. Some questions: - Is this bug a dangerous one or can I ignore it safely? perl complains about it because you may get bogus results from various operations. That doesn't sound ignorable to me. - Is this a bug related to XTerm? Insufficient data. - Should I set the LC_TYPE and LANG variables in /etc/login.conf? (Is this a clean solution?) Why do you think that would solve the problem? - If I want to get the OpenBSD's default locale (is this C/POSIX or another one?) back what file should I link to whom? (Following Pieter's workaround) Why would you do that instead of simply not setting LC_CTYPE? - What about copying a CL_TYPE file from [0] in to the concerned directory which is listed by perl? The files in cvs are in a human-readable text source format that needs to be 'compiled' into a binary format used by the locale subsystem, so simply copying them would be a Bad Thing. Beyond that, there's the question of whether the locale code in the C library will actually handle the locale data for the *.UTF-8 locales. My guess is that the answer is no and that it simply won't work. After all, if it did, you would expect the OpenBSD developers to have included them in the normal builds and distributions. Making the locale stuff work 100% correctly is quite complicated and is still, as I understand it, a work-in-progress. In the meantime, using the ISO8859-1 locales instead of the UTF-8 locales is still, IIRC, the recommended course. Philip Guenther Thank you very much for this very helpful post. As I have some plan9port programs running on the concerned machine I thought that maybe these are responsible for setting this UTF-8 locale (as Plan 9 is pure
Re: [Perl/locales] Warning about locales
Hi! On Sun, Jun 15, 2008 at 11:46:28AM -0700, F. Caulier wrote: [...] I tried to figure out why this problem occurs and following to that I noticed that this perllocale warning only comes up when dropping a pkg_* directly in xterm. When using screen in an xterm and dropping pkg_* to it everything will work fine. Same for tty shells without X where everything works fine too. I don't know much about this terminal stuff, but if everything beside XTerm works fine, could it be that XTerm itself and not the locales are the problems' source? Maybe XTerm doesn't manage to pass on the locales correctly? I don't get those warnings in xterm directly, either. Try env | egrep 'LANG|LC_' in xterm. That shows your environment settings. Then look for where those variables are set. The locations depend on your shell. /etc/profile, /etc/csh.*, /etc/ksh.kshrc, $HOME/.profile, $HOME/.login, $HOME/.bash*. grep them for LANG and LC_ For me, using the default (i.e. LANG and LC_... unset, which is the same as if they were set to C) works well. Could be screen just filters those settings out. Could be that xterm executes a login shell and screen not or vice versa. Some questions: - Is this bug a dangerous one or can I ignore it safely? Probably not dangerous. - Is this a bug related to XTerm? No. - Should I set the LC_TYPE and LANG variables in /etc/login.conf? (Is this a clean solution?) No. - If I want to get the OpenBSD's default locale (is this C/POSIX or another one?) back what file should I link to whom? (Following Pieter's workaround) No link. Just unset the environment variables in your shell startup files. Use the default /etc/profile etc. Setup your shell startup files in the home directories to *not* change the LANG and LC_... environment variables. - What about copying a CL_TYPE file from [0] in to the concerned directory which is listed by perl? [0] http://www.openbsd.org/cgi-bin/cvsweb/src/share/locale/ctype/?only_with_tag=OPENBSD_4_3_BASE What's CL_TYPE? locate CL_TYPE yielded nothing, even with a source tree and a CVS repository installed. Kind regards, Hannah.
Re: [Perl/locales] Warning about locales
Hi! On Sun, Jun 15, 2008 at 03:26:05PM -0600, Philip Guenther wrote: [...] ... I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: UTF-8 uses variable-length characters while ISO8859-* uses fixed-width (8bit) characters. Giving the locale calls the same data for those two is likely to result in incorrect behavior for all characters 127. Wouldn't it be better to simply not lie and just set the locale to en_US.ISO8859-1? Doesn't work for me either: $ LANG=en_GB.ISO8859-1 perl -e 1 perl: warning: Setting locale failed. [...] $ LANG=en_GB.ISO-8859-1 perl -e 1 perl: warning: Setting locale failed. [...] [...] If nothing obvious sticks out, consider debugging further by checking the environment seen by your .xsession (if you xdm) by adding a line like this: env $HOME/.xsession-env.out to it. Similarly, check the shell's environment by doing something similar from your .profile. Oh, right. grepping .xsession or .xinitrc for LANG and LC_ could help too. I tried to figure out why this problem occurs and following to that I noticed that this perllocale warning only comes up when dropping a pkg_* directly in xterm. When using screen in an xterm and dropping pkg_* to it everything will work fine. Same for tty shells without X where everything works fine too. Windows inside screen inherit their environment from the original screen process. So, how do you start the initial (daemon) screen process? From outside X, before running xinitrc? From your .xinitrc or .xsession? From an xterm? Or it could be a login-shell vs. non-login shell difference. I don't know much about this terminal stuff, but if everything beside XTerm works fine, could it be that XTerm itself and not the locales are the problems' source? Maybe XTerm doesn't manage to pass on the locales correctly? Something is setting LC_CTYPE to an unsupported value. That's the program that needs to be fixed. Or OpenBSD could get more locale support *ducks*. (No, I'm not complaining so much, and I'm currently not up to having enough time and energy for coding it). [...] Kind regards, Hannah.
Re: [Perl/locales] Warning about locales
On Mon, Jun 16, 2008 at 7:00 AM, Hannah Schroeter [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 03:26:05PM -0600, Philip Guenther wrote: ... I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: UTF-8 uses variable-length characters while ISO8859-* uses fixed-width (8bit) characters. Giving the locale calls the same data for those two is likely to result in incorrect behavior for all characters 127. Wouldn't it be better to simply not lie and just set the locale to en_US.ISO8859-1? Doesn't work for me either: $ LANG=en_GB.ISO8859-1 perl -e 1 perl: warning: Setting locale failed. [...] $ LANG=en_GB.ISO-8859-1 perl -e 1 perl: warning: Setting locale failed. [...] The specific locale category involved in the original query was LC_CTYPE: perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. OpenBSD does include LC_CTYPE support for ISO8859-1: $ LC_CTYPE=en_GB.ISO8859-1 perl -e 1 $ Philip Guenther
Re: [Perl/locales] Warning about locales
On Sun, Jun 15, 2008 at 03:26:05PM -0600, Philip Guenther wrote: pOn Sun, Jun 15, 2008 at 12:46 PM, F. Caulier [EMAIL PROTECTED] wrote: --- Pieter Verberne [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 07:20:32AM -0700, F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). So something has set LC_CTYPE to en_US.UTF-8, a locale which is not supported by OpenBSD, which results in the setlocale() call failing, something perl complains about so that you don't blame perl when you get broken results. I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: [...] Most (all?) workarounds are bad ideas I think..
Re: [Perl/locales] Warning about locales
Philip Guenther wrote: On Mon, Jun 16, 2008 at 7:00 AM, Hannah Schroeter [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 03:26:05PM -0600, Philip Guenther wrote: ... I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: UTF-8 uses variable-length characters while ISO8859-* uses fixed-width (8bit) characters. Giving the locale calls the same data for those two is likely to result in incorrect behavior for all characters 127. Wouldn't it be better to simply not lie and just set the locale to en_US.ISO8859-1? Doesn't work for me either: $ LANG=en_GB.ISO8859-1 perl -e 1 perl: warning: Setting locale failed. [...] $ LANG=en_GB.ISO-8859-1 perl -e 1 perl: warning: Setting locale failed. [...] The specific locale category involved in the original query was LC_CTYPE: perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. OpenBSD does include LC_CTYPE support for ISO8859-1: $ LC_CTYPE=en_GB.ISO8859-1 perl -e 1 $ Philip Guenther Fwiw, as said, rebooted the system. Installed (make install clean) some p5-* modules and other modules via cpan No error/warning messages. Didn't change any /etc/logib.conf or ~/.profile from stock Was prepared to prepend installs with env LC_ALL=C (old Linux rembrances, in the first weeks they switched to UTF-8) not even needed. OpenBSD GENERIXCis C by default. Basta! So, there must be a culprit somewhere who did change LC_CTYPE=en_US.UTF-8 (empty directory) but forgot to restore the env as it was found. Bah! Bah! Sure this is a GNU thinghie :-) Culprit somewhere, in my case, the -snapshot (c.45) then some 200 make package (cvs June 13): sorry, can't drill down. Anyway, I was sleeping while the puter was busy.
[Perl/locales] Warning about locales
I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). USUAL/NORMAL OUTPUT FOLLOWS If I do this directly in a ttyCx when Xorg is off, I don't get any error messages. The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. (Obviously I must've changend something without seeing it, but what and where?) If I manually set/export LC_ALL and LANG as C, Perl stops complaining. How can I fix this problem? I'm using OpenBSD-4.3, GENERIC Kernel and pdksh.
Re: [Perl/locales] Warning about locales
On Sun, Jun 15, 2008 at 07:20:32AM -0700, F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). I had the same problem. See http://perldoc.perl.org/perllocale.html . But I havn't used that site for my solution. I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. Me too. Pieter Verberne
Re: [Perl/locales] Warning about locales
F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). USUAL/NORMAL OUTPUT FOLLOWS If I do this directly in a ttyCx when Xorg is off, I don't get any error messages. The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. (Obviously I must've changend something without seeing it, but what and where?) If I manually set/export LC_ALL and LANG as C, Perl stops complaining. How can I fix this problem? I'm using OpenBSD-4.3, GENERIC Kernel and pdksh. Supplementary question, but as we are on the subject: would it be correct to edit /etc/login.conf and add the variables under \:default (a la FreeBSD) now, as /usr/share/locale/en_US.UTF-8 is empty, I guess the fallback would be C is it best to leave LC_TYPE to C or dus a setting to en_US.UTF-8 makes a different behaviour (as setting an option to UTF-8) Will try some Perl modules as soon as I get a chance to re-boot.
Re: [Perl/locales] Warning about locales
--- Pieter Verberne [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 07:20:32AM -0700, F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). I had the same problem. See http://perldoc.perl.org/perllocale.html . But I havn't used that site for my solution. I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. Me too. Pieter Verberne Thanks for your fast reply. I tried to figure out why this problem occurs and following to that I noticed that this perllocale warning only comes up when dropping a pkg_* directly in xterm. When using screen in an xterm and dropping pkg_* to it everything will work fine. Same for tty shells without X where everything works fine too. I don't know much about this terminal stuff, but if everything beside XTerm works fine, could it be that XTerm itself and not the locales are the problems' source? Maybe XTerm doesn't manage to pass on the locales correctly? Some questions: - Is this bug a dangerous one or can I ignore it safely? - Is this a bug related to XTerm? - Should I set the LC_TYPE and LANG variables in /etc/login.conf? (Is this a clean solution?) - If I want to get the OpenBSD's default locale (is this C/POSIX or another one?) back what file should I link to whom? (Following Pieter's workaround) - What about copying a CL_TYPE file from [0] in to the concerned directory which is listed by perl? [0] http://www.openbsd.org/cgi-bin/cvsweb/src/share/locale/ctype/?only_with_tag=OPENBSD_4_3_BASE
Re: [Perl/locales] Warning about locales
pOn Sun, Jun 15, 2008 at 12:46 PM, F. Caulier [EMAIL PROTECTED] wrote: --- Pieter Verberne [EMAIL PROTECTED] wrote: On Sun, Jun 15, 2008 at 07:20:32AM -0700, F. Caulier wrote: I get the following when I (as root and standard user) execute pkg_info, pkg_add or pkg_delete with Xorg on: # pkg_info perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LC_ALL = (unset) LC_CTYPE = en_US.UTF-8, LANG = (unset) are supported and installed on your system. perl: warning: Falling back to standard locale (C). So something has set LC_CTYPE to en_US.UTF-8, a locale which is not supported by OpenBSD, which results in the setlocale() call failing, something perl complains about so that you don't blame perl when you get broken results. ... I found a workaround: # ln -s /usr/share/locale/en_GB.ISO8859-1 /usr/share/locale/en_US.UTF-8 That seems like a really bad idea to me. UTF-8 and ISO8859-1 are fundamentally different: UTF-8 uses variable-length characters while ISO8859-* uses fixed-width (8bit) characters. Giving the locale calls the same data for those two is likely to result in incorrect behavior for all characters 127. Wouldn't it be better to simply not lie and just set the locale to en_US.ISO8859-1? The point is that, I didn't changed anything related locales and I couldn't find any config files where these locales are specified. So I'm wondering why this problem appears if I didn't change anything. Well, have you examined all the programs involved in getting to that shell inside xterm to see if any of them document that they alter the environment? For example, if you use 'uxterm' instead of 'xterm' then you'll see exactly the above behavior, as uxterm is just a script that sets LC_CTYPE=en_US.UTF-8 and then invokes xterm with the -en option. If nothing obvious sticks out, consider debugging further by checking the environment seen by your .xsession (if you xdm) by adding a line like this: env $HOME/.xsession-env.out to it. Similarly, check the shell's environment by doing something similar from your .profile. I tried to figure out why this problem occurs and following to that I noticed that this perllocale warning only comes up when dropping a pkg_* directly in xterm. When using screen in an xterm and dropping pkg_* to it everything will work fine. Same for tty shells without X where everything works fine too. Windows inside screen inherit their environment from the original screen process. So, how do you start the initial (daemon) screen process? From outside X, before running xinitrc? From your .xinitrc or .xsession? From an xterm? I don't know much about this terminal stuff, but if everything beside XTerm works fine, could it be that XTerm itself and not the locales are the problems' source? Maybe XTerm doesn't manage to pass on the locales correctly? Something is setting LC_CTYPE to an unsupported value. That's the program that needs to be fixed. Some questions: - Is this bug a dangerous one or can I ignore it safely? perl complains about it because you may get bogus results from various operations. That doesn't sound ignorable to me. - Is this a bug related to XTerm? Insufficient data. - Should I set the LC_TYPE and LANG variables in /etc/login.conf? (Is this a clean solution?) Why do you think that would solve the problem? - If I want to get the OpenBSD's default locale (is this C/POSIX or another one?) back what file should I link to whom? (Following Pieter's workaround) Why would you do that instead of simply not setting LC_CTYPE? - What about copying a CL_TYPE file from [0] in to the concerned directory which is listed by perl? The files in cvs are in a human-readable text source format that needs to be 'compiled' into a binary format used by the locale subsystem, so simply copying them would be a Bad Thing. Beyond that, there's the question of whether the locale code in the C library will actually handle the locale data for the *.UTF-8 locales. My guess is that the answer is no and that it simply won't work. After all, if it did, you would expect the OpenBSD developers to have included them in the normal builds and distributions. Making the locale stuff work 100% correctly is quite complicated and is still, as I understand it, a work-in-progress. In the meantime, using the ISO8859-1 locales instead of the UTF-8 locales is still, IIRC, the recommended course. Philip Guenther
