Re: vim not seeing many Unicode chars

[Tom Browder]

On Tue, May 18, 2021 at 11:37 IL Ka  wrote
…

Thanks!

-Tom

Re: vim not seeing many Unicode chars

[Tom Browder]

 On Tue, May 18, 2021 at 09:36 Greg Wooledge  wrote:

> On Tue, May 18, 2021 at 09:25:30AM -0500,

…

Thanks, Greg.

-Tom

Re: vim not seeing many Unicode chars [SOLVED]

[Tom Browder]

On Tue, May 18, 2021 at 11:38 Steve Dondley  wrote:

> On 2021-05-18 12:23 PM, Tom Browder wrote:
>
> On Tue, May 18, 2021 at 11:08 Steve Dondley  wrote:
>
>> On 2021-05-18 10:25 AM, Tom Browder wrote:
>>
>> I'm running Debian Buster. Inside a terminal window I can use Emacs and
>> can see and enter Unicode chars.
>>
>> But in the same terminal, when I run vim, I have trouble editing or
>> seeing most Unicode chars above ASCII.
>>
>>
>>
>>
>>
>> Type ":set fileencoding?" from inside vim.
>>
>> What do you see?
>>
>
> I see at the bottom of the window (without the square brackets):
>
> [   fileencoding=  ]
>
> -Tom
>
> OK, that's one issue. In your vimrc file, add "set fileencoding=utf8".
> Restart vim. Your files will be saved with utf8.
>
> Now check "set encoding?"
>
> What does that say?
>
> If it says nothing, add "set encoding=utf8" to the vimrc file. Now vim
> should show the files with utf8 encoding.
>
I did all you suggested, and eventually boiled it down to one and only one
line in my $HOME/.vimrc file:

set encoding=utf8

and all works when I vim the file with utf8 chars! I can see them, delete
them, and enter them.

Thanks so much, Steve. As you said, that's the first issue.

Now I'll start cleaning up the locale mess and use the rest of the
suggestions.

Best regards,

-Tom

Re: vim not seeing many Unicode chars

[deloptes]

Eike Lantzsch ZP6CGE wrote:

> To noboy in particular:
> Yes, that is so and for whatever reasons there might be, it is extremely
> confusing causing frustration, despair and eventually anger.
> It is especially exasperating if one works with several DE which,
> contrary to the underlying distribution, also change their behaviour
> once in awhile.
> I used to have en_US language with German keyboard and SA-paraguaian
> monetary symbols. That is an edge-case which worked fine for many years
> but recently gives me trouble with KDE and also Perl.
> Cheers to everybody on the list

This is why I decided to stick to the continuation of KDE3 now called TDE -
using Cyrillic, German and English - no issues.
I refused to be victim of developers/designers dictatorship.
TDE might not be perfect, but the community wants to have stable desktop.

It could also work with newer KDE if it were possible to find one stable
version and stick to it. However your post and similar are not very
motivational to try newer KDE again soon.

Re: vim not seeing many Unicode chars

[Eike Lantzsch ZP6CGE]

On Dienstag, 18. Mai 2021 13:04:00 -04 hdv@gmail wrote:
> On 2021-05-18 18:37, IL Ka wrote:
> > Thanks all. I looked at my config files (which go back at least
> > 15
> > years) and found lots of explicitly setting both LC_ALL=C and
> > LC_LANG=C.
> >
> > Should I remove all, or just remove the LC_ALL?
> >  >
> >  > Using LC_ALL is strongly discouraged as it overrides everything.
> >
> > Please use it only when testing and never set it in a startup file.
> > https://wiki.debian.org/Locale 
> >
> > It is better to set everything explicitly in "/etc/default/locale"
> >
> > Even better: use
> > $ sudo dpkg-reconfigure locales
> > it will generate all locales you need AND set the default one in
> > this file.
> >
> > There is NO reason to use anything except UTF-8 in 2021
>
> Please, be aware that not all Desktop Environments (like KDE's Plasma)
> honour this setting. Often, they have their own mechanisms for
> setting the locale within the DE itself.
>
> Grx HdV

To noboy in particular:
Yes, that is so and for whatever reasons there might be, it is extremely
confusing causing frustration, despair and eventually anger.
It is especially exasperating if one works with several DE which,
contrary to the underlying distribution, also change their behaviour
once in awhile.
I used to have en_US language with German keyboard and SA-paraguaian
monetary symbols. That is an edge-case which worked fine for many years
but recently gives me trouble with KDE and also Perl.
 Cheers to everybody on the list

--
Eike Lantzsch ZP6CGE

Re: vim not seeing many Unicode chars

[Greg Wooledge]

On Tue, May 18, 2021 at 11:24:39AM -0500, Tom Browder wrote:
> Thanks all. I looked at my config files (which go back at least 15 years)
> and found lots of explicitly setting both LC_ALL=C and LC_LANG=C.
> 
> Should I remove all, or just remove the LC_ALL?

LC_LANG isn't even a variable that actually gets used.  Maybe it's a typo
that you made which had no observable negative effects, so it just hung
around.

I would remove both, for sure.

Re: vim not seeing many Unicode chars

[hdv@gmail]

On 2021-05-18 18:37, IL Ka wrote:


Thanks all. I looked at my config files (which go back at least 15
years) and found lots of explicitly setting both LC_ALL=C and LC_LANG=C.

Should I remove all, or just remove the LC_ALL?


 > Using LC_ALL is strongly discouraged as it overrides everything. 
Please use it only when testing and never set it in a startup file.

https://wiki.debian.org/Locale 

It is better to set everything explicitly in "/etc/default/locale"

Even better: use
$ sudo dpkg-reconfigure locales
it will generate all locales you need AND set the default one in this file.

There is NO reason to use anything except UTF-8 in 2021


Please, be aware that not all Desktop Environments (like KDE's Plasma) 
honour this setting. Often, they have their own mechanisms for setting 
the locale within the DE itself.


Grx HdV

Re: vim not seeing many Unicode chars

[Steve Dondley]

On 2021-05-18 12:23 PM, Tom Browder wrote:

> On Tue, May 18, 2021 at 11:08 Steve Dondley  wrote: 
> 
> On 2021-05-18 10:25 AM, Tom Browder wrote: 
> 
> I'm running Debian Buster. Inside a terminal window I can use Emacs and can 
> see and enter Unicode chars. 
> 
> But in the same terminal, when I run vim, I have trouble editing or seeing 
> most Unicode chars above ASCII. 
> 
> Type ":set fileencoding?" from inside vim. 
> 
> What do you see?

I see at the bottom of the window (without the square brackets): 

[   fileencoding=  ] 

-Tom 

OK, that's one issue. In your vimrc file, add "set fileencoding=utf8".
Restart vim. Your files will be saved with utf8. 

Now check "set encoding?" 

What does that say? 

If it says nothing, add "set encoding=utf8" to the vimrc file. Now vim
should show the files with utf8 encoding.

Re: vim not seeing many Unicode chars

[IL Ka]

>
>
> Thanks all. I looked at my config files (which go back at least 15 years)
> and found lots of explicitly setting both LC_ALL=C and LC_LANG=C.
>
> Should I remove all, or just remove the LC_ALL?
>

> Using LC_ALL is strongly discouraged as it overrides everything. Please
use it only when testing and never set it in a startup file.
https://wiki.debian.org/Locale

It is better to set everything explicitly in "/etc/default/locale"

Even better: use
$ sudo dpkg-reconfigure locales
it will generate all locales you need AND set the default one in this file.

There is NO reason to use anything except UTF-8 in 2021




>

Re: vim not seeing many Unicode chars

[Tom Browder]

On Tue, May 18, 2021 at 09:25 Tom Browder  wrote:

> I'm running Debian Buster. Inside a terminal window I can use Emacs and
> can see and enter Unicode chars.
>
...

Thanks all. I looked at my config files (which go back at least 15 years)
and found lots of explicitly setting both LC_ALL=C and LC_LANG=C.

Should I remove all, or just remove the LC_ALL?

-Tom

Re: vim not seeing many Unicode chars

[Tom Browder]

On Tue, May 18, 2021 at 11:08 Steve Dondley  wrote:

> On 2021-05-18 10:25 AM, Tom Browder wrote:
>
> I'm running Debian Buster. Inside a terminal window I can use Emacs and
> can see and enter Unicode chars.
>
> But in the same terminal, when I run vim, I have trouble editing or seeing
> most Unicode chars above ASCII.
>
>
>
>
>
> Type ":set fileencoding?" from inside vim.
>
> What do you see?
>

I see at the bottom of the window (without the square brackets):

[   fileencoding=  ]

-Tom

Re: vim not seeing many Unicode chars

[Steve Dondley]

On 2021-05-18 10:25 AM, Tom Browder wrote:

> I'm running Debian Buster. Inside a terminal window I can use Emacs and can 
> see and enter Unicode chars. 
> 
> But in the same terminal, when I run vim, I have trouble editing or seeing 
> most Unicode chars above ASCII.

Type ":set fileencoding?" from inside vim. 

What do you see?

Re: vim not seeing many Unicode chars

[Tom Browder]

On Tue, May 18, 2021 at 09:25 Tom Browder  wrote:
...

>
I show the LANG env var set to 'en-US.UTF-8'. When I execute 'locale' I get:
>
> LANG=en-US.UTF-8
>

Sorry, typo, should be: 'en_US.UTF-8' # underscore, not hyphen

-Tom

Re: vim not seeing many Unicode chars

[Greg Wooledge]

On Tue, May 18, 2021 at 05:32:26PM +0300, IL Ka wrote:
> >
> > LC_CTYPE="C"
> >
> > this may be a problem.
> 
> Try
> $ LC_CTYPE=en_US.UTF-8 vim $YOUR_FILE

That may not be enough.  The "C" being in quotes in the output of locale
for this variable means it's implicitly defined from another variable,
either LANG or LC_ALL.  In this case, it's coming from LC_ALL because
we saw LC_ALL=C without quotes.

So, overriding LC_ALL would work, but that's just fighting fire with fire,
which is not a long-term solution.

Re: vim not seeing many Unicode chars

[Greg Wooledge]

On Tue, May 18, 2021 at 09:25:30AM -0500, Tom Browder wrote:
> I show the LANG env var set to 'en-US.UTF-8'. When I execute 'locale' I get:
> 
> LANG=en-US.UTF-8
> LANGUAGE=
> LC_CTYPE="C"
> # all intermediate keys: LC_X="C"
> LC_ALL=C

Somewhere, you have placed LC_ALL=C in your environment.  You'll need to
track that down and get rid of it.

Overriding specific variables like LC_TIME or LC_COLLATE is fine, but
you should not be setting LC_ALL on a permanent basis.  It's for short-term
emergency overrides only.

LC_ALL is overriding your LANG variable, giving you the results that you
see, and don't want.  For now, you could unset it, and that would fix the
problem in this one terminal.  But you'll need to figure out where it came
from in order to fix it permanently.

If you use bash as a login shell, one of the ways you can track down where
a variable comes from in your interactive shell is:

PS4='+ $BASH_SOURCE:$FUNCNAME:$LINENO:' bash -ilxc : 2>&1 | grep LC_ALL

This only works if it's coming from a file that's sourced by the shell,
not if it's coming from PAM, or from GNOME, or any other desktop
environment.

(And it assumes your dot file configuration is relatively sane; if for
example you failed to source .bashrc from your .profile or .bash_profile,
then you might need to repeat it with -ixc instead of -ilxc to pick up
the lines from .bashrc as well.  But if you did things correctly, this
shouldn't be needed.)

Re: vim not seeing many Unicode chars

[IL Ka]

>
> LC_CTYPE="C"
>
> this may be a problem.

Try
$ LC_CTYPE=en_US.UTF-8 vim $YOUR_FILE

make sure fileenconfig is utf-8
in vim:
: set fileencoding=utf-8

 More info about LC_ vars:
https://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html


>

vim not seeing many Unicode chars

[Tom Browder]

I'm running Debian Buster. Inside a terminal window I can use Emacs and can
see and enter Unicode chars.

But in the same terminal, when I run vim, I have trouble editing or seeing
most Unicode chars above ASCII.

I show the LANG env var set to 'en-US.UTF-8'. When I execute 'locale' I get:

LANG=en-US.UTF-8
LANGUAGE=
LC_CTYPE="C"
# all intermediate keys: LC_X="C"
LC_ALL=C

My $HOME/.vimrc file is empty.

Note I am a minimalist vim user and use (and rember) very few vim commands.

I appreciate any help.

Best regards,

-Tom