Re: the stupid encoding problem to stdout
Ian Kelly wrote:
> If you want your output to behave that way, then all you have to do is
> specify that with an explicit encode step.
ok
>> If we want we change default for whatever we want, but without this
>> "default change" Python should not change his behavior depending on
>> output. yeah I prefer strange output for a different platform, to a
>> decode errors.
>
> Sorry, I disagree. If your program is going to fail, it's better that
> it fail noisily (with an error) than silently (with no notice that
> anything is wrong).
Hi,
ok a little resume, I got the solution which is setting env with
PYTHONIOENCODING=utf-8, which if it was a default for modern GNU Linux, was
made me save lots of time.
My practical problem is simple like, I make a script that want run in shell
for testing and log to a file when use with a configuration.
Everything runs well in a shell and sometimes (later) fails when log to a
file, with a "UnicodeEncodeError: 'ascii' codec can't encode character
u'\xe7' in position".
So to work in both cases (tty and files), I filled all code with string
.encode('utf-8') to workaround, when what always I want was use
PYTHONIOCONDIG=utf-8. I got anything in utf-8, database is in utf-8, I
coding in utf-8, my OS is in utf-8. In last about 3 years of learning Python
I lost many many hours to understand this problem.
And see, I can send ascii and utf-8 to utf-8 output and never have problems,
but if I send ascii and utf-8 to ascii files sometimes got encode errors.
So you please consider, at least on Linux, default encode to utf-8 (because
we have less problems) or make more clear that pipe to a file is different
to a tty and problem was in files that defaults to ascii. Or
make the default of IOENCONDIG based on env LANG.
Anyway many thanks for your time and for help me out.
I don't know how run the things in Python 3 , in python 3 defaults are utf-8
?
Thanks,
--
Sérgio M. B.
--
http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Ben Finney wrote: >> > What should it decode to, then? >> >> UTF-8, as in tty > > But when you explicitly redirect to a file, it's not going to a TTY. > It's going to a file whose encoding isn't known unless you specify it. ok after thinking about this, this problem exist because Python want be smart with ttys, which is in my point of view is wrong, should not encode to utf-8, because tty is in utf-8. Python should always encode to the same thing. If the default is ascii, should always encode to ascii. yeah should send to tty in ascii, if I send my code to a guy in windows which use tty with cp1000whatever , shouldn't give decoding errors and should send in ascii . If we want we change default for whatever we want, but without this "default change" Python should not change his behavior depending on output. yeah I prefer strange output for a different platform, to a decode errors. And I have /usr/bin/iconv . Thanks for attention, sorry about my very limited English. -- Sérgio M. B. -- http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Ben Finney wrote: >> >> Exactly the opposite , if python don't know the encoding should not >> >> try decode to ASCII. > > Are you advocating that Python should refuse to write characters unless > the encoding is specified? I could sympathise with that, but currently > that's not what Python does; instead it defaults to the ASCII codec. could be a solution ;) or a smarter default based on LANG for example (as many GNU does). -- Sérgio M. B. -- http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Mark Tolonen wrote: > > "Sérgio Monteiro Basto" wrote in message > news:4df137a7$0$30580$a729d...@news.telepac.pt... > >> How I change sys.stdout.encoding always to UTF-8 ? at least have a >> consistent sys.stdout.encoding > > There is an environment variable that can force Python I/O to be a specfic > encoding: > > PYTHONIOENCODING=utf-8 Excellent thanks , double thanks. BTW: should be set by default on a utf-8 systems like Fedora, Ubuntu, Debian , Redhat, and all Linuxs. For sure I will put this on startup of my systems. > -Mark -- Sérgio M. B. -- http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Nobody wrote: >> Exactly the opposite , if python don't know the encoding should not try >> decode to ASCII. > > What should it decode to, then? UTF-8, as in tty, how I change this default ? > You can't write characters to a stream, only bytes. > ok got the point . Thanks, -- http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Ben Finney wrote: > Sérgio Monteiro Basto writes: > >> ./test.py >> moçambique >> moçambique > > In this case your terminal is reporting its encoding to Python, and it's > capable of taking the UTF-8 data that you send to it in both cases. > >> ./test.py > output.txt >> Traceback (most recent call last): >> File "./test.py", line 5, in >> print u >> UnicodeEncodeError: 'ascii' codec can't encode character >> u'\xe7' in position 2: ordinal not in range(128) > > In this case your shell has no preference for the encoding (since you're > redirecting output to a file). > How I say to python that I want that write in utf-8 to files ? > In the first print statement you specify the encoding UTF-8, which is > capable of encoding the characters. > > In the second print statement you haven't specified any encoding, so the > default ASCII encoding is used. > > > Moral of the tale: Make sure an encoding is specified whenever data > steps between bytes and characters. > >> Don't seems logic, when send things to a file the beaviour change. > > They're different files, which have been opened with different > encodings. If you want a different encoding, you need to specify that. > -- http://mail.python.org/mailman/listinfo/python-list
Re: the stupid encoding problem to stdout
Benjamin Kaplan wrote:
> 2011/6/8 Sérgio Monteiro Basto :
>> hi,
>> cat test.py
>> #!/usr/bin/env python
>> #-*- coding: utf-8 -*-
>> u = u'moçambique'
>> print u.encode("utf-8")
>> print u
>>
>> chmod +x test.py
>> ./test.py
>> moçambique
>> moçambique
>>
>> ./test.py > output.txt
>> Traceback (most recent call last):
>> File "./test.py", line 5, in
>> print u
>> UnicodeEncodeError: 'ascii' codec can't encode character
>> u'\xe7' in position 2: ordinal not in range(128)
>>
>> in python 2.7
>> how I explain to python to send the same thing to stdout and
>> the file output.txt ?
>>
>> Don't seems logic, when send things to a file the beaviour
>> change.
>>
>> Thanks,
>> Sérgio M. B.
>
> That's not a terminal vs file thing. It's a "file that declares it's
> encoding" vs a "file that doesn't declare it's encoding" thing. Your
> terminal declares that it is UTF-8. So when you print a Unicode string
> to your terminal, Python knows that it's supposed to turn it into
> UTF-8. When you pipe the output to a file, that file doesn't declare
> an encoding. So rather than guess which encoding you want, Python
> defaults to the lowest common denominator: ASCII. If you want
> something to be a particular encoding, you have to encode it yourself.
Exactly the opposite , if python don't know the encoding should not try
decode to ASCII.
>
> You have a couple of choices on how to make it work:
> 1) Play dumb and always encode as UTF-8. This would look really weird
> if someone tried running your program in a terminal with a CP-847
> encoding (like cmd.exe on at least the US version of Windows), but it
> would never crash.
I want python don't care about encoding terminal and send characters as they
are or for a file .
> 2) Check sys.stdout.encoding. If it's ascii, then encode your unicode
> string in the string-escape encoding, which substitutes the escape
> sequence in for all non-ASCII characters.
How I change sys.stdout.encoding always to UTF-8 ? at least have a
consistent sys.stdout.encoding
> 3) Check to see if sys.stdout.isatty() and have different behavior for
> terminals vs files. If you're on a terminal that doesn't declare its
> encoding, encoding it as UTF-8 probably won't help. If you're writing
> to a file, that might be what you want to do.
Thanks,
--
http://mail.python.org/mailman/listinfo/python-list
the stupid encoding problem to stdout
hi,
cat test.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
u = u'moçambique'
print u.encode("utf-8")
print u
chmod +x test.py
./test.py
moçambique
moçambique
./test.py > output.txt
Traceback (most recent call last):
File "./test.py", line 5, in
print u
UnicodeEncodeError: 'ascii' codec can't encode character
u'\xe7' in position 2: ordinal not in range(128)
in python 2.7
how I explain to python to send the same thing to stdout and
the file output.txt ?
Don't seems logic, when send things to a file the beaviour
change.
Thanks,
Sérgio M. B.
--
http://mail.python.org/mailman/listinfo/python-list
Re: lxml 2.2.4 for Python 2.6
Hi, Srijit Kumar Bhadra wrote: > Is there any reason why lxml-2.2.4-py2.6-win32.egg (md5) or > lxml-2.2.4.win32-py2.6.exe is not available? > > Best regards, > /Srijit maybe ask on lxml Mailing List , should be more appropriated Sérgio M. B. -- http://mail.python.org/mailman/listinfo/python-list
Re: [repost please help me] python setup.py build for 32-bits on x86_64 machine
Diez B. Roggisch wrote:
> Sérgio Monteiro Basto schrieb:
>> Diez B. Roggisch wrote:
>>
>> Hi, Thanks,
>>> Sérgio Monteiro Basto wrote:
>>>
>>>> Hi,
>>>> I am in x86_64 arch , but I need
>>>> compile things on 32 bits with
>>>> python setup.py build
>>>>
>>>> Can't change the fact that distutils creates x86_64
>>>> directories:
>>>> build/temp.linux-x86_64-2.3/
>>>>
>>>> Also if I try with a python compile in 32bits and installed
>>>> in system .
>>> I doubt that. Distutils will always build based on the architecture of
>>> the interpreter you used when building an external module.
>>>
>>> Are you sure that the python you used to build the extension was the
>>> right one? What does
>>>
>>> -c "from distutils.util import get_platform; print
>>> get_platform()"
>> python32 -c "from distutils.util import get_platform; print
>> get_platform()" linux-x86_64
>>
>> ldd ~/bin/python32
>>linux-gate.so.1 => (0xe000)
>>libpthread.so.0 => /lib/libpthread.so.0 (0x00326000)
>>libdl.so.2 => /lib/libdl.so.2 (0x0033f000)
>>libutil.so.1 => /lib/libutil.so.1 (0x006b3000)
>>libm.so.6 => /lib/libm.so.6 (0x00345000)
>>libc.so.6 => /lib/libc.so.6 (0x001e)
>>/lib/ld-linux.so.2 (0x001c2000)
>>
>> this a python 2.3, that's make any difference ?
>
> Ok, next try:
>
> import distutils
> print distutils.sysconfig.get_config_var('SIZEOF_VOID_P')
>
> What does that yield?
>>> import distutils.sysconfig
>>> print distutils.sysconfig.get_config_var('SIZEOF_VOID_P')
None
Thanks, I found the problem, python doesn't have devel stuff like
config/Makefile. A complete python3.5 with config/Makefile solved the
problem .
"distutils gets its compiler options from Python's Makefile", and I don't
have config/Makefile correctly build ...
Big thanks,
--
http://mail.python.org/mailman/listinfo/python-list
Re: [repost please help me] python setup.py build for 32-bits on x86_64 machine
Diez B. Roggisch wrote: Hi, Thanks, > Sérgio Monteiro Basto wrote: > >> Hi, >> I am in x86_64 arch , but I need >> compile things on 32 bits with >> python setup.py build >> >> Can't change the fact that distutils creates x86_64 >> directories: >> build/temp.linux-x86_64-2.3/ >> >> Also if I try with a python compile in 32bits and installed >> in system . > > I doubt that. Distutils will always build based on the architecture of the > interpreter you used when building an external module. > > Are you sure that the python you used to build the extension was the right > one? What does > > -c "from distutils.util import get_platform; print > get_platform()" python32 -c "from distutils.util import get_platform; print get_platform()" linux-x86_64 ldd ~/bin/python32 linux-gate.so.1 => (0xe000) libpthread.so.0 => /lib/libpthread.so.0 (0x00326000) libdl.so.2 => /lib/libdl.so.2 (0x0033f000) libutil.so.1 => /lib/libutil.so.1 (0x006b3000) libm.so.6 => /lib/libm.so.6 (0x00345000) libc.so.6 => /lib/libc.so.6 (0x001e) /lib/ld-linux.so.2 (0x001c2000) this a python 2.3, that's make any difference ? > > return? > > Diez -- http://mail.python.org/mailman/listinfo/python-list
[repost please help me] python setup.py build for 32-bits on x86_64 machine
Hi, I am in x86_64 arch , but I need compile things on 32 bits with python setup.py build Can't change the fact that distutils creates x86_64 directories: build/temp.linux-x86_64-2.3/ Also if I try with a python compile in 32bits and installed in system . how I force distutils build to 32-bits on x86_64 arch? Thanks in advance, -- http://mail.python.org/mailman/listinfo/python-list
python setup.py build 32-bits on x86_64 machine
Hi, I am in x86_64 arch , but I need compile things on 32 bits. python setup.py build Can't change the fact that distutils creates x86_64 directories: gcc -pthread -shared build/temp.linux-x86_64-2.3/ Also if I try with a python compile in 32bits and installed in system . how I force distutils build to 32-bits ? Thanks in advance, -- http://mail.python.org/mailman/listinfo/python-list
Re: HTMLParser.HTMLParseError: EOF in middle of construct
Stefan Behnel wrote:
> Sérgio Monteiro Basto wrote:
>> but is one single error that blocks this.
>> Finally I found it , it is :
>> > if I put :
>> >
>> p = re.compile('"align')
>> content = p.sub('" align', content)
>>
>> I can parse the html
>> I don't know if it a bug of HTMLParser
>
> Sure, and next time your key doesn't open your neighbours house, please
> report to the building company to have them fix the door.
>
The question, here, is if
http://mail.python.org/mailman/listinfo/python-list
Re: HTMLParser.HTMLParseError: EOF in middle of construct
First, sorry about the mess, let see if kontact works better with newsgroups. Marc 'BlackJack' Rintsch wrote: >> ok but my problem is not understand what is the specific problem at line >> 1173 > > You can't just look at that line and ignore the rest. There are 604 (!) > errors, some about table rows, before this line. So the parser may be > confused at this point and be already in an internal state that sees that > line in a completely different light than you do. Ok , but is one single error that blocks this. Finally I found it , it is : http://mail.python.org/mailman/listinfo/python-list
