Re: [O] org-babel, python, encoding and table
On 2013-05-29 22:00, Eric Schulte wrote:
This does seem to be in the right direction. This is similar to my
diagnosis of the non-session case [1]. If the python process prints
weird hex character values (e.g., '\xc3\xa0') instead of utf8 characters
(e.g., "à"), then Babel will insert the hex values into the buffer.
As I recall switching to python3 solved this problem?
What I do regularily in my python code to force python2 into emitting utf8
characters is the following:
#+BEGIN_SRC python
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
#+END_SRC
There are other methods, for example resetting the default encoding in the
site-wide file site.py, but that is not very portable.
Hope this helps,
Christian
--
Christian Wittern, Kyoto
Re: [O] org-babel, python, encoding and table
>
> To remind and illustrate the problem, here is an example :
> #+name: pytab-test
> #+begin_src python :results value :session :preamble # -*- coding: utf-8 -*-
> :return a
> a = ( ( "é", "a" ), ( "a", "à" ) )
> a
> #+end_src
>
> #+TBLNAME: pytab-test
> | \303\251 | a|
> | a| \303\240 |
>
>
> I have then two problems :
> 1. The characters are not well displayed in the buffer
> 2. If I try to save the buffer, emacs doesn't recognize the encoding and
> tells me that "utf-8-unix cannot encode these: \303 \251 [...]
>
> So I decided to inspect what happened during the Python session...
> Basically, Org-babel just write the str conversion of my tuple ( ( "é", "a"
> ), ( "a", "à" ) ) (that appears (('\xc3\xa9',
> 'a'), ('a', '\xc3\xa0')) in the python interpreter) in a temporary file.
>
> Then looking in this temporary file, I see that the strange characters
> are written directly \xc3, \xa9, etc.
>
> Consequently, my guess is that org-babel has maybe some difficulties
> to deal with these characters while reading the temporary file before
> displaying the results in the buffer.
>
> Unfortunately, this is just a guess and even less a solution... But am I on
> relevant lead???
>
This does seem to be in the right direction. This is similar to my
diagnosis of the non-session case [1]. If the python process prints
weird hex character values (e.g., '\xc3\xa0') instead of utf8 characters
(e.g., "à"), then Babel will insert the hex values into the buffer.
As I recall switching to python3 solved this problem?
For the session case, it may be possible to fix this by changing the
python code in the `org-babel-python-evaluate-session' function around
line 300 in ob-python.el.
Cheers,
>
> Thanks in advance for any help...
>
> Roland.
>
>
>
>
Footnotes:
[1] http://article.gmane.org/gmane.emacs.orgmode/71866
--
Eric Schulte
http://cs.unm.edu/~eschulte
Re: [O] org-babel, python, encoding and table
Andreas Röhler easy-emacs.de> writes:
>
> Am 07.05.2013 18:41, schrieb Eric Schulte:
> >> #+NAME: test2
> >> #+begin_src python :results value :preamble # -*- coding: utf-8 -*-
:return
> >> a
> >> a = ( ( "é", "a" ), ( "a", "à" ) )
> >> b = "é"
> >> #+end_src
> >>
> >> #+RESULTS: test2
> >> | \303\251 | a|
> >> | a| \303\240 |
> >>
> >
> > Maybe this isn't an execution problem, but is rather a buffer encoding
> > problem. I executed your example above in a small buffer (attached). I
> > then saved this buffer and was forced to specify an encoding, I selected
> > utf8. If I cat the resulting file from disk, the accented characters
> > appear correctly.
> >
>
> Confirming this.
>
> BTW also return a[0][0] displays correct so far.
>
> Cheers,
>
> Andreas
>
>
Hello,
Just an update about this post.
I've kept on digging on the problem of org-babel python results that
produces encoding problems in
the emacs buffer when the requested results is turned into a org table.
To remind and illustrate the problem, here is an example :
#+name: pytab-test
#+begin_src python :results value :session :preamble # -*- coding: utf-8 -*-
:return a
a = ( ( "é", "a" ), ( "a", "à" ) )
a
#+end_src
#+TBLNAME: pytab-test
| \303\251 | a|
| a| \303\240 |
I have then two problems :
1. The characters are not well displayed in the buffer
2. If I try to save the buffer, emacs doesn't recognize the encoding and
tells me that "utf-8-unix cannot encode these: \303 \251 [...]
So I decided to inspect what happened during the Python session...
Basically, Org-babel just write the str conversion of my tuple ( ( "é", "a"
), ( "a", "à" ) ) (that appears (('\xc3\xa9',
'a'), ('a', '\xc3\xa0')) in the python interpreter) in a temporary file.
Then looking in this temporary file, I see that the strange characters are
written directly \xc3, \xa9, etc.
Consequently, my guess is that org-babel has maybe some difficulties to deal
with these characters
while reading the temporary file before displaying the results in the
buffer.
Unfortunately, this is just a guess and even less a solution... But am I on
relevant lead???
Thanks in advance for any help...
Roland.
Re: [O] org-babel, python, encoding and table
Am 07.05.2013 18:41, schrieb Eric Schulte: #+NAME: test2 #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: test2 | \303\251 | a| | a| \303\240 | Maybe this isn't an execution problem, but is rather a buffer encoding problem. I executed your example above in a small buffer (attached). I then saved this buffer and was forced to specify an encoding, I selected utf8. If I cat the resulting file from disk, the accented characters appear correctly. Confirming this. BTW also return a[0][0] displays correct so far. Cheers, Andreas
Re: [O] org-babel, python, encoding and table
Am 07.05.2013 22:05, schrieb Nick Dokos: Andreas Röhler writes: So it's up to choose the Python default interpreter. Unfortunatly (org-babel-do-load-languages 'org-babel-load-languages '((python3 . t))) seems not working. The language is still python (more or less). What you need to do is customize org-babel-python-command. Thanks. This reveals a bug in ob-python then. Will open a separate thread. Andreas
Re: [O] org-babel, python, encoding and table
Andreas Röhler writes: > So it's up to choose the Python default interpreter. > > Unfortunatly > > (org-babel-do-load-languages > 'org-babel-load-languages > '((python3 . t))) > > seems not working. > The language is still python (more or less). What you need to do is customize org-babel-python-command. > while > > type python3 > python3 ist /usr/bin/python3 > -- Nick
Re: [O] org-babel, python, encoding and table
[ ... ] But, what is hurting my poor little neurone is the fact that I don't understand why I get the correct result when the evaluation returns a single value and a wrong answer with a table. that's not the case. One from returns a, the other b, which are bound differently. The tuple fails with Python2 that way. I think I have to understand how org-babel builds a table from a Python list of lists but I have to confess that I'm afraid of reading Lisp programs, though Emacs is my best friend ;)... Well, I'll keep on investigating on that subject Thanks again for your help! Roland.
Re: [O] org-babel, python, encoding and table
Nick Dokos gmail.com> writes: > > Andreas Röhler easy-emacs.de> writes: > > > Am 07.05.2013 20:18, schrieb Eric Schulte: > >> Andreas Röhler easy-emacs.de> writes: > >> ... > >> Maybe Python simply needs to be convinced to print in utf-8 format? > > > > Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3 > > unicode handling is one of the big changes between Python 2.x and Python > 3.x. It's good to know that Python 3.x seems to make it trivial to > handle unicode correctly (although there might still be dragons there). > > Here are some links: > > http://docs.python.org/2/howto/unicode.html > http://docs.python.org/3/howto/unicode.html > > that might shed some light. > Thank you very much for your answers!!! Well, I read that Python 3.x has improved the encoding handling but for now I have to keep using 2.7.x Python version. Using Python outside of org-mode is not a problem to manage encoding correctly. In general, you can use encode/decode methods to get what you want. Then, using the print function turns the coded character into human readable ones. But, what is hurting my poor little neurone is the fact that I don't understand why I get the correct result when the evaluation returns a single value and a wrong answer with a table. I think I have to understand how org-babel builds a table from a Python list of lists but I have to confess that I'm afraid of reading Lisp programs, though Emacs is my best friend ;)... Well, I'll keep on investigating on that subject Thanks again for your help! Roland.
Re: [O] org-babel, python, encoding and table
Am 07.05.2013 21:20, schrieb Nick Dokos: Andreas Röhler writes: Am 07.05.2013 20:18, schrieb Eric Schulte: Andreas Röhler writes: ... Maybe Python simply needs to be convinced to print in utf-8 format? Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3 unicode handling is one of the big changes between Python 2.x and Python 3.x. It's good to know that Python 3.x seems to make it trivial to handle unicode correctly (although there might still be dragons there). Here are some links: http://docs.python.org/2/howto/unicode.html http://docs.python.org/3/howto/unicode.html that might shed some light. So it's up to choose the Python default interpreter. Unfortunatly (org-babel-do-load-languages 'org-babel-load-languages '((python3 . t))) seems not working. while type python3 python3 ist /usr/bin/python3
Re: [O] org-babel, python, encoding and table
Andreas Röhler writes: > Am 07.05.2013 20:18, schrieb Eric Schulte: >> Andreas Röhler writes: >> ... >> Maybe Python simply needs to be convinced to print in utf-8 format? > > Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and > Python3.3 - all called from Emacs24.3 unicode handling is one of the big changes between Python 2.x and Python 3.x. It's good to know that Python 3.x seems to make it trivial to handle unicode correctly (although there might still be dragons there). Here are some links: http://docs.python.org/2/howto/unicode.html http://docs.python.org/3/howto/unicode.html that might shed some light. -- Nick
Re: [O] org-babel, python, encoding and table
Am 07.05.2013 20:18, schrieb Eric Schulte: Andreas Röhler writes: Am 07.05.2013 18:41, schrieb Eric Schulte: #+NAME: test2 #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: test2 | \303\251 | a| | a| \303\240 | Maybe this isn't an execution problem, but is rather a buffer encoding problem. I executed your example above in a small buffer (attached). I then saved this buffer and was forced to specify an encoding, I selected utf8. If I cat the resulting file from disk, the accented characters appear correctly. So maybe the problem is simply Emacs not displaying utf8 characters correctly. Hope this helps, Here error occurs depending if a or b is specified for return, really strange. I think this may be a problem with the Python code. The snippet above expands to the following python. which, when executed returns the following raw string (which is then interpreted by Org-mode). Maybe Python simply needs to be convinced to print in utf-8 format? Andreas Get the wrong results with a Ipython0.12, but correct with Python3.2.3 and Python3.3 - all called from Emacs24.3 <>
Re: [O] org-babel, python, encoding and table
Andreas Röhler writes:
> Am 07.05.2013 18:41, schrieb Eric Schulte:
>>> #+NAME: test2
>>> #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return
>>> a
>>> a = ( ( "é", "a" ), ( "a", "à" ) )
>>> b = "é"
>>> #+end_src
>>>
>>> #+RESULTS: test2
>>> | \303\251 | a|
>>> | a| \303\240 |
>>>
>>
>> Maybe this isn't an execution problem, but is rather a buffer encoding
>> problem. I executed your example above in a small buffer (attached). I
>> then saved this buffer and was forced to specify an encoding, I selected
>> utf8. If I cat the resulting file from disk, the accented characters
>> appear correctly.
>>
>>
>>
>>
>> So maybe the problem is simply Emacs not displaying utf8 characters
>> correctly.
>>
>> Hope this helps,
>>
>
> Here error occurs depending if a or b is specified for return, really strange.
>
I think this may be a problem with the Python code. The snippet above
expands to the following python.
# -*- coding: utf-8 -*-
def main():
a = ( ( "é", "a" ), ( "a", "à" ) )
b = "é"
return a
open('/tmp/output', 'w').write( str(main()) )
which, when executed returns the following raw string (which is then
interpreted by Org-mode).
(('\xc3\xa9', 'a'), ('a', '\xc3\xa0'))
Maybe Python simply needs to be convinced to print in utf-8 format?
>
> Andreas
>
>
--
Eric Schulte
http://cs.unm.edu/~eschulte
Re: [O] org-babel, python, encoding and table
Am 07.05.2013 18:41, schrieb Eric Schulte: #+NAME: test2 #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: test2 | \303\251 | a| | a| \303\240 | Maybe this isn't an execution problem, but is rather a buffer encoding problem. I executed your example above in a small buffer (attached). I then saved this buffer and was forced to specify an encoding, I selected utf8. If I cat the resulting file from disk, the accented characters appear correctly. So maybe the problem is simply Emacs not displaying utf8 characters correctly. Hope this helps, Here error occurs depending if a or b is specified for return, really strange. Andreas
Re: [O] org-babel, python, encoding and table
> #+NAME: test2 > #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return > a > a = ( ( "é", "a" ), ( "a", "à" ) ) > b = "é" > #+end_src > > #+RESULTS: test2 > | \303\251 | a| > | a| \303\240 | > Maybe this isn't an execution problem, but is rather a buffer encoding problem. I executed your example above in a small buffer (attached). I then saved this buffer and was forced to specify an encoding, I selected utf8. If I cat the resulting file from disk, the accented characters appear correctly. #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: | é | a| | a| à | So maybe the problem is simply Emacs not displaying utf8 characters correctly. Hope this helps, -- Eric Schulte http://cs.unm.edu/~eschulte
[O] org-babel, python, encoding and table
Hello, My problem is about python code evaluation with org-babel that should give a table containing accented characters. Here is an example : #+NAME: test1 #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return b a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: test1 : é It's ok, no problem! But : #+NAME: test2 #+begin_src python :results value :preamble # -*- coding: utf-8 -*- :return a a = ( ( "é", "a" ), ( "a", "à" ) ) b = "é" #+end_src #+RESULTS: test2 | \303\251 | a| | a| \303\240 | I don't understand why the accented characters are replaced by some codes when the results is interpreted as org-table... Any idea, workaround to solve my problem would be much appreciated! I use Org-mode version 8.0.2 (8.0.2-2-g93da18-elpaplus @ /home/roland/.emacs.d/elpa/org-plus-contrib-20130429/). Thanks in advance. Best regards, Roland.
