Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Aahz]

On Thu, Jul 23, 2009, INADA Naoki wrote:
>
> I found WriteConsoleW() API recently.
> This API can write utf16 string to console directly, without change
> OutputCodepage.
> 
> example:
> http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42
> 
> I think this API is good for py3k.
> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

Please submit a feature request to bugs.python.org -- with a patch would
be even nicer, of course.
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

"The volume of a pizza of thickness 'a' and radius 'z' is
given by pi*z*z*a"
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Martin v. Löwis]

> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

I think this would be fairly difficult to implement given
the way the output libraries work. If you think it can be
done, please provide a patch to bugs.python.org.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[INADA Naoki]

Hi.

>> We have to use "IME" to input non-ascii charactor in Windows.
>> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
>>
>> So setting codepage to 65001 make output universal but make input ascii-only.
>> Sit!!!
>
> Is there a code page that still allows IME input, but supports all
> of Unicode? I know there is GB18030 - is it any good?

I found WriteConsoleW() API recently.
This API can write utf16 string to console directly, without change
OutputCodepage.

example:
http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42

I think this API is good for py3k.
When stdout is console and not redirected to [pipe|file],
sys.stdout.write(u"foo")
can avoid encoding and use WriteConsoleW(L"foo")

-- 
Naoki INADA  
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Glenn Linderman]

On approximately 3/24/2009 10:16 AM, came the following characters from 
the keyboard of INADA Naoki:

Hi. I'm Japanese and non-ascii charactor user. (cp932)

We have to use "IME" to input non-ascii charactor in Windows.
When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.

So setting codepage to 65001 make output universal but make input ascii-only.
Sit!!!

I hope PyQtShell  become good
IDLE alternative.



Thanks for the feedback.

So at least one version of the code I posted shows that 
programmatically, the code page can be set differently for input and 
output, although the last version brought both to 65001.  It seems that 
the chcp 65001 always does both.  If the IME only works for cp932, then 
leave input at cp932, and set output to 65001?


I have no idea if that could be a solution for you, but I would be 
interested in your results if you find that it is, or isn't, as that 
would add to the collective knowledge base about the subject.  This is 
idea 2, below, where I tried to cover the solution space more broadly.


Looking briefly at the definition of cp932, it seems that it covers most 
of the Unicode characters... so perhaps any or several of the following 
could happen:


1) the IME could be converted to produce UTF-8 instead of cp932, 
allowing use of 65001 for input and output
2) the split code page could be used to avoid the conversion of Unicode 
to cp932 for output.
3) Unicode could be converted to cp932 for output, allowing use of cp932 
for both input and output.


These are listed in the order of increased overhead for character handling.

Perhaps you could enlighten us all as to the issues with each of these 
ideas.


I realize the IME exists today, and is likely coded to use cp932, and 
that it would take some work to convert it to produce Unicode.  However, 
there seems to be a straightforward conversion chart between cp932 and 
Unicode at Wikipedia, so perhaps that isn't a huge effort.


It seems that the long term goal of having all software speak Unicode 
would increase the efficiency of all software when dealing with 
multi-lingual issues, as a common solution can be applied universally, 
rather than re-inventing solutions that only work for particular code pages.


But I'm not fully aware of whether or not the design or implementation 
of Unicode precludes universal solutions: I have heard rumors that 
certain characters must be interpreted differently in different locale 
contexts, which seems to be counter to the "one solution fits all" 
possibility.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Martin v. Löwis]

> Hi. I'm Japanese and non-ascii charactor user. (cp932)
> 
> We have to use "IME" to input non-ascii charactor in Windows.
> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
> 
> So setting codepage to 65001 make output universal but make input ascii-only.
> Sit!!!

Is there a code page that still allows IME input, but supports all
of Unicode? I know there is GB18030 - is it any good?

In any case, for *that* problem, I think you need to look for a
different terminal program.

> I hope PyQtShell  become good
> IDLE alternative.

Could well be. To see it integrated into Python (and perhaps replace
IDLE), a lot of steps would need to happen, though. But perhaps you
weren't asking for it to be included in Python - it can also be a good
alternative if it's not included (like several other IDEs)

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[INADA Naoki]

Hi. I'm Japanese and non-ascii charactor user. (cp932)

We have to use "IME" to input non-ascii charactor in Windows.
When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.

So setting codepage to 65001 make output universal but make input ascii-only.
Sit!!!

I hope PyQtShell  become good
IDLE alternative.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Glenn Linderman]

On approximately 3/23/2009 12:12 PM, came the following characters from 
the keyboard of Terry Reedy:

Glenn Linderman wrote:



One can set CMD into Unicode mode (chcp 65001)... not sure how Python 
reacts to that either.  But even then...


I tried that and others have reported doing so on python-list but no one 
has gotten that to work.



http://support.microsoft.com/kb/247815
http://www.microsoft.com/downloads/details.aspx?familyid=22e69ae4-7e40-4807-8a86-b3d36fab68d3&displaylang=en

(python 3)

import ctypes
k=ctypes.WinDLL('kernel32')
x = k.SetConsoleOutputCP(65001)
if x!= 1:
print("x was ", x )
exit( 1 )
print (''.join(chr(i) for i in range(0x410, 0x430)).encode('utf-8'))

produces a nice b'\xd0\x90\d0\x91' stream of hex representations of 
UTF-8 encoded Unicode characters...


The only thing that seems to be missing is that Python won't emit them 
to the screen that way.


So surely some python-dev that is smarter than me, could provide that 
magic incantation.  Will go search, but that isn't in my current 
knowledge banks.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Terry Reedy]

Glenn Linderman wrote:



One can set CMD into Unicode mode (chcp 65001)... not sure how Python 
reacts to that either.  But even then...


I tried that and others have reported doing so on python-list but no one 
has gotten that to work.


CMD will only use fixed-width fonts, and none of the standard XP ones 
seem to contain all of Unicode.  Not sure if that has improved on Vista 
or 7, as they don't run here.


It _would_ be nice to get this resolved, somehow.


Definitely.

tjr



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Martin v. Löwis]

> 2) IDLE does much better but its support seems to still be imcomplete.
> Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's
> shell an alternate UI.

That is certainly *no* good SoC project. Instead, just report it as
a *specific* bug report (rather than saying "it seems incomplete").

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Ben Finney]

Terry Reedy  writes:

> If Windows (or other OSes) (to be investigated) does not reliably
> come with a full unicode font (at least current BMP), is there a
> public domain or open license font that we can include?

The GNU Unifont at Unifoundry http://unifoundry.com/> is designed
for this purpose.

-- 
 \  “For my birthday I got a humidifier and a de-humidifier. I put |
  `\  them in the same room and let them fight it out.” —Steven Wright |
_o__)  |
Ben Finney

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Glenn Linderman]

On approximately 3/22/2009 8:48 PM, came the following characters from 
the keyboard of Terry Reedy:
One of the disappointments of CPython 3.0 on Windows is that the switch 
to unicode for text (str), coupled with the continued use of a 
unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means 
that print can no longer print all str strings, or all legal Python code 
(as in a traceback).


Different people have tried and failed fix this bug by fiddling with 
'Command Prompt' configeration.  This would make a useful summer 
project, though I don't know if it would involve enough coding.  Call 
the project something like 3.0 Unicode UI Improvement.


I see two possibilities.

1) Find an C-coded open-source C-P replacement whose author will license 
to PSF and, as needed, modify or integrate it with CPython.


2) IDLE does much better but its support seems to still be imcomplete. 
Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's 
shell an alternate UI.


If Windows (or other OSes) (to be investigated) does not reliably come 
with a full unicode font (at least current BMP), is there a public 
domain or open license font that we can include?



One can, of course, set CMD into Latin-1 mode (chcp 1252)... not sure 
how Python reacts to that, as I've only used it with Perl, until I gave 
up on Perl's Unicode support (which someone finally seems to be fixing, 
but then there is CPAN to improve).  But that doesn't solve the font 
problem, nor characters above 255.


One can set CMD into Unicode mode (chcp 65001)... not sure how Python 
reacts to that either.  But even then...


CMD will only use fixed-width fonts, and none of the standard XP ones 
seem to contain all of Unicode.  Not sure if that has improved on Vista 
or 7, as they don't run here.


It _would_ be nice to get this resolved, somehow.


--
Glenn -- http://nevcal.com/
===
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI

[Terry Reedy]

One of the disappointments of CPython 3.0 on Windows is that the switch 
to unicode for text (str), coupled with the continued use of a 
unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means 
that print can no longer print all str strings, or all legal Python code 
(as in a traceback).


Different people have tried and failed fix this bug by fiddling with 
'Command Prompt' configeration.  This would make a useful summer 
project, though I don't know if it would involve enough coding.  Call 
the project something like 3.0 Unicode UI Improvement.


I see two possibilities.

1) Find an C-coded open-source C-P replacement whose author will license 
to PSF and, as needed, modify or integrate it with CPython.


2) IDLE does much better but its support seems to still be imcomplete. 
Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's 
shell an alternate UI.


If Windows (or other OSes) (to be investigated) does not reliably come 
with a full unicode font (at least current BMP), is there a public 
domain or open license font that we can include?


Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com