Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Paul DeStefano]

Hello Rich & PLUG,

I'm not sure if this topic was completed to Rich's satisfaction, so let me 
add something.


On Tuesday, 26 July 2016, Rich Shepard wrote:
  My virtual terminals use urxvt and correctly display UTF-8 text 
encoding as demonstrated by viewing the UTF-8-demo.txt file. My MUA is 
alpine and invoked in a urxvt terminal. But, not all foreign languages 
are displayed; for example, Hebrew is seen as a series of question 
marks. This is one question I'd like to have answered, if possible.


I've been using PINE since the mid 90's and have run into this problem 
several times.  In 2009, there was another discussion on the alpine-info 
list about it.  The first attached message is from that discussion and 
includes a nice font test I have used many times since for testing.  I'm 
not sure which UTF-8 demo you have.  This one includes Hebrew and works 
for me in Alpine (in gnome-terminal, see below).


You should check out the alpine-info archives; I'm pretty sure the answers 
are there.  But, here is what I have learned.  The problem with UTF-8 
characters is solved for almost all combinations.  For example, I see the 
same characters in xterm using True-Type fonts with both Alpine and less. 
But the remaining problems are not related to UTF-8 encoding but 16-bit or 
"wide" characters commonly used in Chinese, Japanese, and Korean (CJK). 
Maybe Hebrew, too?


I don't receive any messages in Hebrew, but I receive a lot in Japanese. 
The trick is gnome-terminal.  Alpine in gnome-terminal gives the best 
results.  This is because Alpine passes the characters on to the terminal 
(mostly) unmolested and it's the terminal's job to finds the right glyph. 
Most chars even work in regular old xterm if you tell xterm to use nice, 
modern fonts ("liberation mono", for example).  But, only gnome-terminal 
supports *variable width encoding.* (Not to be confused with variable with 
fonts, which regular old xterm supports using TTF.)  So, try that.


A complete "true" solution is probably found by following the CJK users. 
I've investigated this myself once, but gave up since my current set-up is 
adequate.  This, I assume, would work on the console, not just in terminal 
emulators (xterm,urxvt,gnome-terminal,etc.).  I wonder what happens if you 
choose Chinese on a fresh install then use the input-chooser hot-keys to 
switch to English keyboard?


  The second question probably relates to how alpine is configured. Or 
maybe not. When I compose an e-mail message the v.t. changes from urxvt 
to xterm. Might this have something to do with my specifying 'joe' as 
the text editor rather than 'nano'? This is another issue that's not 
critical but a matter of curiosity whether it can be explained.


I would use VIM or Emacs, for sure, but I don't think that is your 
problem.


What do you mean by v.t. "changes" from urxvt to xterm?  For that matter, 
what do you mean by your v.t. "uses" urxvt?  I think I'm missing something 
important here.


In the past, I tried many times so address font problems with urxvt as it 
seems pretty popular.  But, in my testing, xterm is still superior now 
that it supports all the same fonts as, say, libreoffice or the desktop 
environment, itself.


  Alpine is configured to display UTF-8 text and to use /usr/bin/joe as 
the editor. If there are options that address the above two questions 
then I missed them.


I've discussed this with Eduardo Chappa, myself, and there are some 
options that affect character passing/encoding between Alpine and the 
terminal.  "Pass C1 control chars as is" for example, but I don't believe 
that affects this issue.  You already have UTF-8 selected as the display 
char set, so that's the main thing.


--
Paul DeStefanoFrom alpine-(ta)-benizi(dot) com Mon Jan 12 18:55:59 2009
Date: Mon, 12 Jan 2009 21:50:12 -0500 (EST)
From: Benjamin R. Haskell 
To: alpine-i...@u.washington.edu
Subject: Re: [Alpine-info] Foreign Characters?

On Mon, 12 Jan 2009, Dan Mahoney, System Admin wrote:

>>> c) screen/termcap
>
> I found that I had to do ctrl-a :utf8 on in the window running alpine. 
> Screen was still blocking it until I did this.  I now find that the last 
> character of your last name looks (to me) like a regular z, with "rabbit 
> ears" (I don't know the character name).  Is that right?

Yes, Andraž's name ends with a z with a caron (ž):
ž   017eLATIN SMALL LETTER Z WITH CARON

(I've heard it more often called a háček (hacek) 
http://en.wikipedia.org/wiki/Hacek )


> Could someone with another name/text (preferably asian) send something 
> to me to check this with as well?

I've wanted for a very long time to be able to display more than just 
Eurocentric "foreign" characters inside a terminal that I could also use 
for my normal work (in English).  My guess is that your sticking point is 
going to be the font.  Something like Firefox will handle many scripts, 
because it will fall back to a different font depending 

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Paul Heinlein]

On Tue, 26 Jul 2016, Rich Shepard wrote:

> On Tue, 26 Jul 2016, Michael Rasmussen wrote:
>
>> A text console still has font choices. Your system documents how.
>
> Michael,
>
> That's very true. And all my consoles and virtual terminals properly 
> display UTF-8 ... except some languages in Alpine.

On one of my Linux systems (CentOS 7, in case it matters), I can run 
cat or less against a Hebrew text file and it shows correctly.

If I send that same text file to myself as an e-mail attachment, 
Alpine will NOT show -- or even save -- it correctly. (I tried sending 
the file from both a Linux host and from a Mac host; the file was 
garbled in both tests.)

Then, I changed the name of the file from, e.g., Hebrew.txt to 
Hebrew.hb and sent it. Since the mail client wasn't able to recognize 
the .hb suffix, the attachments was marked as type 
'Application/OCTET-STREAM' instead of 'Text/PLAIN.' That made all the 
difference: Alpine was able to save the file correctly. (It still 
didn't display it correctly.)


If I put Hebrew directly into the body of the message, Alpine will 
display and reply to it just fine -- but I'm using the built-in 
pico-esque editor, not an external one.


So my first suspicion is that your text isn't getting the correct MIME 
type, but that's just a guess based on what I'm seeing above.

Beyond that, I don't have time to track down what's going on.

-- 
Paul Heinlein <> heinl...@madboa.com <> http://www.madboa.com/
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Rich Shepard]

On Tue, 26 Jul 2016, Michael Rasmussen wrote:

> A text console still has font choices.
> Your system documents how.

Michael,

   That's very true. And all my consoles and virtual terminals properly
display UTF-8 ... except some languages in Alpine.

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Rich Shepard]

On Tue, 26 Jul 2016, John Jason Jordan wrote:

> Doesn't the terminal use a font for display purposes? It's probably a
> monospaced font but, I thought everything ultimately used some font or
> another.

John,

   Yes it's using the 10x20 monospaced, san-serif font.

> Or do you mean that X isn't even running, e.g., you boot to the command
> line? In my case I used Gnome terminal and in its preferences I see that
> it is using 'System Fixed Width Font.' Just for kicks I grabbed some
> Hebrew text from a web page and pasted into Gnome terminal. It looked
> perfect. But Gnome terminal is a GUI. I never boot without X, so I don't
> know what might happen.

   I always boot into runlevel 3, multiuser console mode, and run starx to
load the X Window System. Sometimes I work in one or more consoles.

   I'm a command line user as much as possible: alpine, emacs, R, etc. As a
touch-typist (thanks to the Army insisting I type at least 55 wpm to get out
of counter-intelligence school), I much prefer to work from the keyboard
rather than rolling the trackball around. As many GUI applications that I
use that can be configured to work from the keyboard and ignore the
pointy-thing are set up that way.

Thanks,

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Michael Rasmussen]

On Tue, Jul 26, 2016 at 09:56:30AM -0700, Rich Shepard wrote:
> On Tue, 26 Jul 2016, John Jason Jordan wrote:
> 
> > Fonts? Which are set as the display fonts for the various programs?
> 
> John,
> 
>Standard 10x20. This is a non-GUI application.

A text console still has font choices. 
Your system documents how.

http://docs.slackware.com/slackware:localization#setting_the_console_font

-- 
  Michael Rasmussen, Portland Oregon  
Be Appropriate && Follow Your Curiosity
Decaffeinated coffee is like a hooker who only wants to cuddle.
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[John Jason Jordan]

On Tue, 26 Jul 2016 09:56:30 -0700 (PDT)
Rich Shepard  dijo:

>On Tue, 26 Jul 2016, John Jason Jordan wrote:
>> Fonts? Which are set as the display fonts for the various programs?

>   Standard 10x20. This is a non-GUI application.

Doesn't the terminal use a font for display purposes? It's probably a
monospaced font but, I thought everything ultimately used some font or
another. Or do you mean that X isn't even running, e.g., you boot to
the command line? In my case I used Gnome terminal and in its
preferences I see that it is using 'System Fixed Width Font.' Just for
kicks I grabbed some Hebrew text from a web page and pasted into Gnome
terminal. It looked perfect. But Gnome terminal is a GUI. I never boot
without X, so I don't know what might happen. 

>> What happens if you copy and paste some Hebrew ?? text into something
>> else, e.g., Firefox, Lyx, etc.?
>
>   Not relevant. I want to see these fonts in alpine.

Yes, that is your goal. I was just struggling for clues. 
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[Rich Shepard]

On Tue, 26 Jul 2016, John Jason Jordan wrote:

> Fonts? Which are set as the display fonts for the various programs?

John,

   Standard 10x20. This is a non-GUI application.

> What happens if you copy and paste some Hebrew ?? text into something
> else, e.g., Firefox, Lyx, etc.?

   Not relevant. I want to see these fonts in alpine.

Thanks,

Rich


___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] UTF-8, Virtual Consoles, and Alpine

[John Jason Jordan]

On Tue, 26 Jul 2016 08:04:23 -0700 (PDT)
Rich Shepard  dijo:

>My virtual terminals use urxvt and correctly display UTF-8 text encoding
>as demonstrated by viewing the UTF-8-demo.txt file. My MUA is alpine
>and invoked in a urxvt terminal. But, not all foreign languages are
>displayed; for example, Hebrew is seen as a series of question marks.
>This is one question I'd like to have answered, if possible.
>
>The second question probably relates to how alpine is configured. Or maybe
>not. When I compose an e-mail message the v.t. changes from urxvt to
>xterm. Might this have something to do with my specifying 'joe' as the
>text editor rather than 'nano'? This is another issue that's not
>critical but a matter of curiosity whether it can be explained.
>
>Alpine is configured to display UTF-8 text and to use /usr/bin/joe as the
>editor. If there are options that address the above two questions then
>I missed them.

Fonts? Which are set as the display fonts for the various programs?
What happens if you copy and paste some Hebrew ?? text into something
else, e.g., Firefox, Lyx, etc.? 
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug